Definition of the given robots.txt file:
User-agent: *
The asterisk (*) or wildcard represents a special value and means any robot.
The Robots text file open in Notepad might look like this:
Disallow:
The Disallow: line without a / (forward slash) tells the robots that they can index the entire site.
Any empty value, indicates that all URLs can be retrieved. At least one Disallow field needs to be present in a record without the / (forward slash) as shown above.
The presence of an empty “/robots.txt” file has no explicit associated semantics, it will be treated as if it was not present, i.e. all robots will consider themselves welcome.
