1. Akshay Masand's Avatar


    Developer Jan Moesen recently discovered that Apple appears to be crawling HTML websites with its own automated bot but the exact purpose of the mysterious homegrown software continues to remain unknown. Moesen found hits from the software dating back to October 15. Others seem to have noticed a large amount of traffic to their websites from Apple’s mysterious crawler which appears to be shut off from time to time.

    Apple’s bot only appears to request for HTML pages and doesn’t access the accompanying CSS, JavaScript or image files from a website. Moesen went on to find that the Apple web crawler identifies itself as “Mozilla/5.0 (compatible; Fetcher/0.1)”. This crawler was found originating from an IP address that started with “17.”, which indicates that the bot is running from Apple’s servers.

    Not much was known about the bot. As of right now we don’t even know if the Apple bot is an official company project or if an employee built and ran the system unofficially from a company machine. There is a chance that Apple’s efforts could be related to Spotlight enhancements in OS X Yosemite and iOS 8. Especially considering both were updated this fall with improvements that allow users to access Web search content directly without the need to access a third-party search tool.

    Although Apple doesn’t seem to compete directly with Google or Microsoft, the Cupertino California company has been working to provide specific information directly within Siri search results as opposed to having to rely on other search engines. This additional research could potentially make Siri less reliant on the current official Siri Web search partner, Bing.

    We’ll have to wait and see what the deal really is once more information surfaces.

    Source: Jan Moesen (Twitter) via AppleInsider

    Twitter: @AkshayMasand
    2014-11-07 05:49 AM
  2. edwilk55's Avatar
    17.0.0.0/8...well at least we have it narrowed down. LOL!
    2014-11-07 01:25 PM
  3. Jahooba's Avatar
    Whatever it is it's not malicious, so no big deal.

    I'd like to know why the "Request Desktop Version" option in iOS 8 doesn't always work. Can websites override that and refuse the request from mobile handsets?
    2014-11-07 05:50 PM
  4. hybirdd's Avatar
    Whatever it is it's not malicious, so no big deal.

    I'd like to know why the "Request Desktop Version" option in iOS 8 doesn't always work. Can websites override that and refuse the request from mobile handsets?
    Some websites redirect to a different subdomain for mobile users so requesting a desktop site may not work on those sites.
    Or some sites may have a responsive design and will always display a mobile like version.
    2014-11-07 06:01 PM
  5. Funken Ferret's Avatar
    Some websites redirect to a different subdomain for mobile users so requesting a desktop site may not work on those sites.
    Or some sites may have a responsive design and will always display a mobile like version.
    2014-11-07 08:53 PM
LINK TO POST COPIED TO CLIPBOARD