Release notes#
1.4.1 (2023-02-10)#
Fixed#
Encode the
FEEDS
command-line argument as JSON.
1.4.0 (2023-02-07)#
Added#
Add
item_url
andlog_url
to the response from the listjobs.json webservice. (@mxdev88)Scrapy 2.8 support. Scrapyd sets
LOG_FILE
andFEEDS
command-line arguments, instead ofSCRAPY_LOG_FILE
andSCRAPY_FEED_URI
environment variables.Python 3.11 support.
Python 3.12 support. Use
packaging.version.Version
instead ofdistutils.LooseVersion
. (@pawelmhm)
Changed#
Rename environment variables to avoid spurious Scrapy deprecation warnings.
SCRAPY_EGG_VERSION
toSCRAPYD_EGG_VERSION
SCRAPY_FEED_URI
toSCRAPYD_FEED_URI
SCRAPY_JOB
toSCRAPYD_JOB
SCRAPY_LOG_FILE
toSCRAPYD_LOG_FILE
SCRAPY_SLOT
toSCRAPYD_SLOT
SCRAPY_SPIDER
toSCRAPYD_SPIDER
Attention
These are undocumented and unused, and may be removed in future versions. If you use these environment variables, please report your use in an issue.
Removed#
Scrapy 1.x support.
Python 3.6 support.
Unmaintained files (Debian packaging) and unused code (
scrapyd/script.py
).
Fixed#
Print Scrapyd’s version instead of Twisted’s version with
--version
(-v
) flag. (@niuguy)Override Scrapy’s
LOG_STDOUT
setting toFalse
to suppress logging output for listspiders.json webservice. (@Lucioric2000)
1.3.0 (2022-01-12)#
Added#
Add support for HTTP authentication.
Make
project
argument to listjobs.json optional, to easily query for all jobs.Improve HTTP headers across webservices.
Add shortcut to jobs page to cancel a job using the cancel.json webservice.
Add configuration options for job storage class and egg storage class.
Improve test coverage.
Python 3.7, 3.8, 3.9, 3.10 support.
Removed#
Python 2, 3.3, 3.4, 3.5 support.
PyPy 2 support.
Documentation for Ubuntu installs (Zyte no longer maintains the Ubuntu package).
Fixed#
Respect Scrapy’s
TWISTED_REACTOR
setting.Replace deprecated
SafeConfigParser
withConfigParser
.
1.2.1 (2019-06-17)#
Fixed#
Fix HTTP header types for newer Twisted versions.
DeferredQueue
no longer hides a pending job when reachingmax_proc
.AddVersion
’s arguments’ string types no longer break Windows environments.test: Update binary eggs to be compatible with Scrapy 1.x.
Removed#
Remove deprecated SQLite utilities.
1.2.0 (2017-04-12)#
Added#
Webservice
Add daemonstatus.json webservice.
Add project version argument to the schedule.json webservice.
Add jobid argument to the schedule.json webservice.
Add the run’s PID to the response of the listjobs.json webservice.
Include full tracebacks from Scrapy when failing to get spider list. This makes debugging deployment problems easier, but webservice output noisier.
Website
Add
webroot
configuration option for website root class.Add start and finish times to jobs page.
Make console script executable.
Add contributing documentation.
Twisted 16 support.
Python 3 support.
Changed#
Change
bind_address
default to 127.0.0.1, instead of 0.0.0.0, to listen only for connections from localhost.
Removed#
Deprecate unused SQLite utilities in the
scrapyd.sqlite
module.SqliteDict
SqlitePickleDict
SqlitePriorityQueue
PickleSqlitePriorityQueue
Scrapy 0.x support.
Python 2.6 support.
Fixed#
Poller race condition for concurrently accessed queues.
1.1.1 (2016-11-03)#
Added#
Document and include missing configuration options in
default_scrapyd.conf
.Document the spider queue’s
priority
argument.Enable some missing tests for the SQLite queues.
Removed#
Disable bdist_wheel command in setup to define dynamic requirements, despite pip-7 wheel caching bug.
Fixed#
Use correct type adapter for sqlite3 blobs. In some systems, a wrong type adapter leads to incorrect buffer reads/writes.
FEED_URI
was always overridden by Scrapyd.Specify maximum versions for requirements that became incompatible.
Mark package as zip-unsafe because Twistd requires a plain
txapp.py
.
1.1.0 (2015-06-29)#
Features & Enhancements#
Outsource scrapyd-deploy command to scrapyd-client (c1358dc, c9d66ca..191353e) If you rely on this command, install the scrapyd-client package from pypi.
Look for a
~/.scrapyd.conf
file in the users home (1fce99b)Adding the nodename to identify the process that is working on the job (fac3a5c..4aebe1c)
Allow remote items store (e261591..35a21db)
Debian sysvinit script (a54193a, ff457a9)
Add ‘start_time’ field in webservice for running jobs (6712af9, acd460b)
Check if a spider exists before schedule it (with sqlite cache) (#8, 288afef..a185ff2)
Bugfixes#
F̶i̶x̶ ̶s̶c̶r̶a̶p̶y̶d̶-̶d̶e̶p̶l̶o̶y̶ ̶-̶-̶l̶i̶s̶t̶-̶p̶r̶o̶j̶e̶c̶t̶s̶ ̶(̶9̶4̶2̶a̶1̶b̶2̶)̶ → moved to scrapyd-client
Sanitize version names when creating egg paths (8023720)
Copy txweb/JsonResource from scrapy which no longer provides it (99ea920)
Use w3lib to generate correct feed uris (9a88ea5)
Fix GIT versioning for projects without annotated tags (e91dcf4 #34)
Correcting HTML tags in scrapyd website monitor (da5664f, 26089cd)
Fix FEED_URI path on windows (4f0060a)
Setup script and Tests/CI#
Restore integration test script (66de25d)
Changed scripts to be installed using entry_points (b670f5e)
Renovate scrapy upstart job (d130770)
Travis.yml: remove deprecated
--use-mirros
pip option (b3cdc61)Mark package as zip unsafe because twistd requires a plain
txapp.py
(f27c054)Removed python 2.6/lucid env from travis (5277755)
Made Scrapyd package name lowercase (1adfc31)
Documentation#
Spiders should allow for arbitrary keyword arguments (696154)
Various typos (51f1d69, 0a4a77a)
Fix release notes: 1.0 is already released (6c8dcfb)
Point website module’s links to readthedocs (215c700)
Remove reference to ‘scrapy server’ command (f599b60)
1.0.2 (2016-03-28)#
setup script#
Specified maximum versions for requirements that became incompatible.
Marked package as zip-unsafe because twistd requires a plain
txapp.py
documentation#
Updated broken links, references to wrong versions and scrapy
Warn that scrapyd 1.0 felling out of support
1.0.1 (2013-09-02)#
Trivial update
1.0.0 (2013-09-02)#
First standalone release (it was previously shipped with Scrapy until Scrapy 0.16).