566E
首页 包列表 提交包 常见问题 关于

承接 fastwhale/phpscraper 相关项目开发

从需求分析到上线部署，全程专人跟进，保证项目质量与交付效率

邮箱：yvsm@zunyunkeji.com | QQ：316430983 | 微信：yvsm316

fastwhale/phpscraper

Composer 安装命令：

composer require fastwhale/phpscraper

包简介

PHPScraper, built with simplicity in mind. See tests/ for more examples.

关键字：

#PHP Library #php crawler #web scraping #PHP scraper #PHP scraping #web-access #xpath scraper

README 文档

README

PHP Scraper: a web utility for PHP

Unit Tests Total Downloads Latest Version License

For full documentation, visit phpscraper.de.

PHPScraper is a versatile web-utility for PHP. Its primary objective is to streamline the process of extracting information from websites, allowing you to focus on accomplishing tasks without getting caught up in the complexities of selectors, data structure preparation, and conversion.

Under the hood, it uses

See composer.json for more details.

⏲️ PHPScraper in 5 Minutes explained

Here are a few impressions of the way the library works. More examples are on the project website.

Basics: Flexible Calling as an Attribute or Method

All scraping functionality can be accessed either as a function call or a property call. For example, the title can be accessed in two ways:

// Prep
$web = new \Fastwhale\PHPScraper\PHPScraper;
$web->go('https://google.com');

// Returns "Google"
echo $web->title;

// Also returns "Google"
echo $web->title();

🔋 Batteries included: Meta data, Links, Images, Headings, Content, Keywords, ...

Many common use cases are covered already. You can find prepared extractors for various HTML tags, including interesting attributes. You can filter and combine these to your needs. In some cases there is an option to get a simple or detailed version, here in the case of linksWithDetails:

$web = new \Fastwhale\PHPScraper\PHPScraper;

// Contains:
// <a href="https://placekitten.com/456/500" rel="ugc">
//   <img src="https://placekitten.com/456/400">
//   <img src="https://placekitten.com/456/300">
// </a>
$web->go('https://test-pages.phpscraper.de/links/image-urls.html');

// Get the first link on the page and print the result
print_r($web->linksWithDetails[0]);
// [
//     'url' => 'https://placekitten.com/456/500',
//     'protocol' => 'https',
//     'text' => '',
//     'title' => null,
//     'target' => null,
//     'rel' => 'ugc',
//     'image' => [
//         'https://placekitten.com/456/400',
//         'https://placekitten.com/456/300'
//     ],
//     'isNofollow' => false,
//     'isUGC' => true,
//     'isSponsored' => false,
//     'isMe' => false,
//     'isNoopener' => false,
//     'isNoreferrer' => false,
// ]

If there aren't any matching elements (here links) on the page, an empty array will be returned. If a method normally returns a string it might return null. Details such as follow_redirects, etc. are optional configuration parameters (see below).

Most of the DOM should be covered using these methods:

A full list of methods with example code can be found on phpscraper.de. Further examples are in the tests.

Download Files

Besides processing the content on the page itself, you can download files using fetchAsset:

// Absolute URL
$csvString = $web->fetchAsset('https://test-pages.phpscraper.de/test.csv');

// Relative URL after navigation
$csvString = $web
  ->go('https://test-pages.phpscraper.de/meta/lorem-ipsum.html')
  ->fetchAsset('/test.csv');

You will only need to write the content into a file or cloud storage.

Process the RSS feeds, sitemap.xml, etc.

PHPScraper can assist in collecting feeds such as RSS feeds, sitemap.xml-entries and static search indexes. This can be useful when deciding on the next page to crawl or building up a list of pages on a website.

Here we are processing the sitemap into a set of FeedEntry-DTOs:

(new \Fastwhale\PHPScraper\PHPScraper)
    ->go('https://phpscraper.de')
    ->sitemap

// array(131) {
//   [0]=>
//   object(Fastwhale\PHPScraper\DataTransferObjects\FeedEntry)#165 (3) {
//     ["title"]=>
//     string(0) ""
//     ["description"]=>
//     string(0) ""
//     ["link"]=>
//     string(22) "https://phpscraper.de/"
//   }
//   [1]=>
// ...

Whenever post-processing is applied, you can fall back to the underlying *Raw-methods.

Process CSV-, XML- and JSON files and URLs

PHPScraper comes out of the box with file / URL processing methods for CSV-, XML- and JSON:

  • parseJson
  • parseXml
  • parseCsv
  • parseCsvWithHeader (generates an asso. array using the first row)

Each method can process both strings as well as URLs:

// Parse JSON into array:
$json = $web->parseJson('[{"title": "PHP Scraper: a web utility for PHP", "url": "https://phpscraper.de"}]');
// [
//     'title' => 'PHP Scraper: a web utility for PHP',
//     'url' => 'https://phpscraper.de'
// ]

// Fetch and parse CSV into a simple array:
$csv = $web->parseCsv('https://test-pages.phpscraper.de/test.csv');
// [
//     ['date', 'value'],
//     ['1945-02-06', 4.20],
//     ['1952-03-11', 42],
// ]

// Fetch and parse CSV with first row as header into an asso. array structure:
$csv = $web->parseCsvWithHeader('https://test-pages.phpscraper.de/test.csv');
// [
//     ['date' => '1945-02-06', 'value' => 4.20],
//     ['date' => '1952-03-11', 'value' => 42],
// ]

Additional CSV parsing parameters such as separator, enclosure and escape are possible.

There is more!

There are plenty of examples on the PHPScraper website and in the tests.

Check the playground.php if you prefer learning by doing. You get it up and running with:

$ git clone git@github.com:spekulatius/PHPScraper.git && composer update

💪 Roadmap

The future development is organized into milestones. Releases follow semver.

v1: Building the first stable version

  • Improve documentation and examples.
  • Organize code better (move websites into separate repos, etc.)
  • Add support for feeds and some typical file types.

v2: Service Upgrade:

v3: Expand the functionality and cover more 'types'

  • Expand to parse a wider range of types, elements, embeds, etc.
  • Improve performance with caching and concurrent fetching of assets
  • Minor improvements for parsing methods

v4: Expand to provide more guidance on building custom scrapers on top of PHPScraper

TBC.

😍 Sponsors

PHPScraper is sponsored by:

With your support, PHPScraper can became the PHP swiss army knife for the web. If you find PHPScraper useful to your work, please consider a sponsorship or donation. Thank you 💪

⚙️ Configuration (optional)

If needed, you can use the following configuration options:

User Agent

You can set the browser agent using setConfig:

$web->setConfig([
  'agent' => 'Mozilla/5.0 (X11; Linux x86_64; rv:107.0) Gecko/20100101 Firefox/107.0'
]);

It defaults to Mozilla/5.0 (compatible; PHP Scraper/1.x; +https://phpscraper.de).

Proxy Support

You can configure proxy support with setConfig:

$web->setConfig(['proxy' => 'http://user:password@127.0.0.1:3128']);

Timeout

You can set the timeout using setConfig:

$web->setConfig(['timeout' => 15]);

Setting the timeout to zero will disable it.

Disabling SSL

While unrecommended, it might be required to disable SSL checks. You can do so using:

$web->setConfig(['disable_ssl' => true]);

You can call setConfig multiple times. It stores the config and merges it with previous settings. This should be kept in mind in the unlikely use-case when unsetting values.

🚀 Installation with Composer

composer require spekulatius/phpscraper

After the installation, the package will be picked up by the Composer autoloader. If you are using a common PHP application or framework such as Laravel or Symfony you can start scraping now 🚀

If not or you are building a standalone-scraper, please include the autoloader in vendor/ at the top of your file:

<?php

require __DIR__ . '/vendor/autoload.php';

// ...

Now you can now use any of the examples on the documentation website or from the tests/-folder.

Please consider supporting PHPScraper with a star or sponsorship:

composer thanks

Thank you 💪

✅ Testing

The library comes with a PHPUnit test suite. To run the tests, run the following command from the project folder:

composer test

You can find the tests here. The test pages are publicly available.

MISC: Issues, Ideas, Contributing, CHANGELOG, UPGRADING, LICENSE

fastwhale/phpscraper 适用场景与选型建议

fastwhale/phpscraper 是一款 基于 PHP 开发的 Composer 扩展包，目前已累计 30 次下载、GitHub Stars 达 0， 最近一次更新时间为 2024 年 12 月 09 日， 在 PHP 生态内属于活跃度较高的组件。

它主要适用于以下技术方向： 「PHP Library」 「php crawler」 「web scraping」 「PHP scraper」 「PHP scraping」 「web-access」 等业务场景。在实际项目中，围绕这些方向常见需要落地的问题包括：接口对接、性能调优、并发安全、与既有框架（Laravel / ThinkPHP / Yii / Webman 等）的兼容适配，以及生产环境的日志埋点与稳定性保障。

我们在过去多个企业项目中使用过 fastwhale/phpscraper 或与其功能相近的方案，如果你在选型或落地过程中遇到问题，例如 版本兼容、二次改造、私有化封装、与内部系统对接、生产 BUG 排查，欢迎联系我们协助评估。

围绕 fastwhale/phpscraper 我们能提供哪些服务？
定制开发 / 二次开发

基于 fastwhale/phpscraper 在你已有业务上做功能扩展、字段裁剪、UI 适配、与内部账号 / 权限 / 日志系统的深度对接。

BUG 修复 & 性能优化

线上偶发问题、内存泄漏、慢查询、并发异常等排查修复；针对高流量场景做缓存、队列、索引层面的调优。

项目外包 & 长期维护

承接完整的项目从需求 → 设计 → 开发 → 上线 → 长期运维；也可按月提供技术保姆服务。

yvsm@zunyunkeji.com QQ：316430983 微信：yvsm316 西安尊云信息科技 · 专注 PHP / Go / 分布式系统研发

与 fastwhale/phpscraper 相关的其它包

同方向 / 同关键字的高下载量 PHP Composer 包推荐，方便对比选型：

eddieace/php-simple 13

crawlbase/crawlbase 16

A lightweight, dependency free PHP class that acts as wrapper for Crawlbase API

migliori/sitemap-crawler 3

Sitemap crawler/generator. For the given URL it will return sitemap XML file with URLs and images.

shipsaas/laravel-inbox-process 13

Inbox pattern process implementation for your Laravel Applications

intahwebz/core 0

Core library that defines common interfaces used by the rest of the intahwebz..

cast/amqp 0

Amqp classes

统计信息

  • 总下载量： 30
  • 月度下载量： 0
  • 日度下载量： 0
  • 收藏数： 0
  • 点击次数： 19
  • 依赖项目数： 0
  • 推荐数： 0

GitHub 信息

  • Stars： 0
  • Watchers： 0
  • Forks： 75
  • 开发语言： PHP
访问仓库

其他信息

  • 授权协议： GPL-3.0-or-later
  • 更新时间： 2024-12-09