MitsuWiki

Cette page est en lecture seule. Vous pouvez afficher le texte source, mais ne pourrez pas le modifier. Contactez votre administrateur si vous pensez qu'il s'agit d'une erreur.
====== Magnetik NyaaTorrents ======
===== Lightweight Infohash Database =====

----
<file>IMPORTANT: LiVID/Magnetik is no longer developed ! Using it in a production environment is strongly unwise and difficult.</file>

**The new project is called [[php:magnyaa|Magnyaa]]. Don't waste more time on this page ;-)**

----

**Download:**

  - {{:php:livid_r1.zip|LiVID r1}} (TPB)
  - {{:php:livid_r2-nyaa.zip|LiVID r2}}
  - {{:php:magnetik_r3-nyaa1.zip|Magnetik r3}}
  - {{:php:magnetik_r4.zip|Magnetik r4}}  {{:php:store.db.7z|store.db}}
  - {{:php:magnetik_r5.zip|Magnetik r5}}  {{:php:data_31641-86312.7z|data (31641-86312)}}  {{:php:nyaa_feeder.zip|bash "feeders"}}

**release 5 - "You Wouldn't Steal A Lion" **

  * MAJOR REWRITING, NOT BACKWARDS-COMPATIBLE !
  * code portage on fSQL (flat file storage, no more SQLite)
  * one table per 10 000 id (nyaaid 261 381 would be in table nyaa26, you see ?)
  * added download count as reported by Nyaa
  * massive script rewriting, cross-script requiring
  * improved XML, CSV and RSS export
  * minor tweaks on search (Google link) and details (magnet explanation as abbr)
  * light CSS improvements
  * OFFSET not supported as of 10th June: listing navigation not possible -- reported to fSQL dev -> the related functions have been commented out
  * add.php fusion with update.php. First try to INSERT, then SELECT and UPDATE. No more duplicate entries and more easy update.
  * name sometimes messing things up: now storage with mysql_real_escape_string, and data transmission as urlencode/urldecode
  * update.php returns status (if success, echo:  nyaaid::btih )
  * pretty much everything depending on table (latest, or provided)


**release 4 - "WTF Am I Reading" **

  * MAJOR REWRITING, NOT BACKWARDS-COMPATIBLE !
  * torrent information now depending on NyaaID
  * database structure changed: "ref" becomes "info", "track" becomes "stat"
  * info table: nyaaid, cat, date, submitter, btih, name, tname, size
  * stat table: nyaaid, updated, seed, leech
  * introducing NyaaID storage (torrent id on Nyaa), category storage (categories from Nyaa), submit date (date of submission of torrent on Nyaa), submitter (username on Nyaa)
  * add.php handles more parameters
  * fulldb.php and search.php show NyaaID, category, download button, seed/leech-count.
  * index.php shows the new things of the database
  * search.php now refuses any search string below 3 characters, and returns a maximum of 100 results (most recent)
  * NEW: rss.php (last 100)
  * NEW: fulldb.php with modes: "nyaaid" for nyaaids, "btih" for BTIH, "xml" for a XML database dump and "csv" for a csv database dump
  * torrent detail now has link back to Nyaa
  * fulldb.php has a "jump to page X" function
  * fulldb.php lists in decending order (most recent at page 1)
  * search.php and index.php adapted to search for name or submitter

  * minor CSS tweaks

**release 3 - "Pissing Code Like I Mean It" **

  * moar comments !
  * added fixes to insert "bizarre" torrent names correctly
  * name transmission, stored separately from torrent name (make sure to replace "&" by something else like "and" in the GET request, or the name will be cut - take a look at the bash feeders)
  * limited output, pagination and counts for fulldb.php
  * information update (update.php)
  * countermeasure against "database locked" (busyTimeout lets it breathe up to 10 seconds)
  * config.php now including footer (to easily add tracking javascripts, or stuff)

**release 2 - "Nyaa~" **

  * fork for NyaaTorrents (download and parsing of torrent files)
  * database structure simplified

**release 1 - "Do What You Want 'Cause A Pirate Is Free" **

  * first public release, working HTML, insertion, search and full listing.

===  how Magnetik and [[php:magnesia|Magnesia]] were born ===


In the 23th, February 2012, some time after Megaupload's shutdown, many concerns appeared about The Pirate Bay (for example, DNS censoring in UK). While the P2P structure allows the data to remain even if the website is shut down, the BitTorrent technology is lacking a real search function. BitTorrent should be as decentralized as possible, and therefore the usage of .torrent files should be avoided (the connections being established with the DHT).

While better solutions are integrated in the BitTorrent clients (like the gossip-protocol search of Tribler), one can simply host a "little Pirate Bay" that, in combination with the search engines, should provide many many different sources from where getting the most important information about a torrent (it's name, btih and size - well we can add the seed- and leechcount, giving a hint of the torrent's popularity).

I started working with a full-PHP system, the torrent data being stored in infohash-named PHP files. Well it IS working, but created thousands of little files in the db/ subfolder.
Manipulating a real database ? Well ok, but it shouldn't rely on a heavyweight like MySQL or PostGreSQL (not very easy to backup and restore). So I started messing around with SQLite3 and fSQL. And here we are :)

SQLite and fSQL are working well and all, but for very heavy databases they just aren't fast enough. We're talking about retrieving little data and parsing mid-long listings: it's much more efficient to split the data and load it on-demand as close as possible from the target. This leads us to [[php:magnesia|Magnesia]]. We talk about it later..

===  SQLite3 ? fSQL ? Does it work on my hosting ? ===
 
Better check out before you get started ! My secondary host supports both SQLite2 and SQLite3, it didn't helped me when I wrote the queries. Hopefully it's 100% SQLite3 and has maximal compatibility.
Take a look at phpinfo() if SQLite3 is enabled on your hosting. If something's wrong, please report and suggest fixes :-)

fSQL, while supporting MySQL-like commands, is still a PHP script. Having PHP5 installed is much all you need :-)

===  [[php:magnesia|Magnesia]] doesn't rely on SQLite/fSQL ? ===

As said: for heavy databases we need something faster, much faster, but still portable. For Magnesia my choice is a plain file storage, in PHP files, the data being stored and loaded as arrays with a requiring as needed/asked. So no worry about millions of entries, PHP only loads the needed files and doesn't have to parse more than 1000 arrays. And it does it quite well and fast, on both PHP4 and PHP5. Simply said: it works fucking good on almost every web hosting for a potential heavy database (according to my calculations, the full The Pirate Bay database wouldn't weigh more than 2 GB and any data from it could still be called very fast).

===  installation and setup ===

Different releases work differently. The most reliable way to set up things right is to read the code ;-) Some releases feature the bash scripts. Magnesia's database is solely built with the bash scripts.
 
- customize "config.php" (you may want to change the title, description, the CSS, the table structure... and the addition password !)
- upload
- begin populating your database

-> make sure the data is correctly formatted before "wgetting" it to the instance ! The data retrieval can be affected by changes, especially on NyaaTorrents: maybe you'll have to modify the awk functions to get the right data out of the page.

===  LEGAL NOTICE ===

This software is provided "as-is", without any warranties.
The author of this software doesn't encourage counterfeit content distribution.
The author cannot be held liable of any misuse of this software. The code is neutral, only a human can do illegal things with it.
It is the user's sole responsibility for downloading and sharing illegal content.
The information returned by the database can be inaccurate, obsolete or plain fake: automated processes can't always succeed.
To comply with some countries' laws, the hostmaster can be asked to remove manually any information about an obviously illegal torrent in the database.


===  LICENSE ===

Magnetik Nyaatorrents, published under [[http://creativecommons.org/licenses/by-sa/3.0/fr/|Creative Commons by-sa]] by Mitsu.

* included: torrent-rw, published under GNU GPL by Adrien Gibrat: https://github.com/adriengibrat/torrent-rw/

* included: fSQL, published under GNU LGPL by Steve "the_unknown": https://sourceforge.net/projects/fsql/

===  CONTACT ===

Bug report, feature requests, love declarations ? Head over here:

http://www.suumitsu.eu/contact