mirror of
https://github.com/dariusk/rss-to-activitypub.git
synced 2024-11-24 02:09:19 +02:00
First
This commit is contained in:
Darius Kazemi
commit
36bb506bcb
|
|
@ -0,0 +1,4 @@
|
|||
node_modules/
|
||||
*.db
|
||||
package-lock.json
|
||||
config.json
|
||||
|
|
@ -0,0 +1,22 @@
|
|||
Copyright (c) 2018 Darius Kazemi
|
||||
|
||||
Permission is hereby granted, free of charge, to any person
|
||||
obtaining a copy of this software and associated documentation
|
||||
files (the "Software"), to deal in the Software without
|
||||
restriction, including without limitation the rights to use,
|
||||
copy, modify, merge, publish, distribute, sublicense, and/or sell
|
||||
copies of the Software, and to permit persons to whom the
|
||||
Software is furnished to do so, subject to the following
|
||||
conditions:
|
||||
|
||||
The above copyright notice and this permission notice shall be
|
||||
included in all copies or substantial portions of the Software.
|
||||
|
||||
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
|
||||
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES
|
||||
OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
|
||||
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT
|
||||
HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
|
||||
WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
|
||||
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR
|
||||
OTHER DEALINGS IN THE SOFTWARE.
|
||||
|
|
@ -0,0 +1,80 @@
|
|||
# RSS to ActivityPub Converter
|
||||
|
||||
This is a server that lets users convert any RSS feed to an ActivityPub actor that can be followed by users on ActivityPub-compliant social networks like Mastodon. For a demo of this in action, see https://bots.tinysubversions.com/convert/
|
||||
|
||||
## Requirements
|
||||
|
||||
This requires Node.js v10.10.0 or above.
|
||||
|
||||
## Installation
|
||||
|
||||
Clone the repository, then `cd` into its root directory. Install dependencies:
|
||||
|
||||
`npm i`
|
||||
|
||||
Then copy `config.json.template` to `config.json`:
|
||||
|
||||
`cp config.json.template config.json`
|
||||
|
||||
Update your new `config.json` file:
|
||||
|
||||
```js
|
||||
{
|
||||
"DOMAIN": "mydomain.com",
|
||||
"PORT_HTTP": "3000",
|
||||
"PORT_HTTPS": "8443",
|
||||
"PRIVKEY_PATH": "/path/to/your/ssl/privkey.pem",
|
||||
"CERT_PATH": "/path/to/your/ssl/cert.pem"
|
||||
}
|
||||
```
|
||||
|
||||
`DOMAIN`: your domain! this should be a discoverable domain of some kind like "example.com" or "rss.example.com"
|
||||
`PORT_HTTP`: the http port that Express runs on
|
||||
`PORT_HTTPS`: the https port that Express runs on
|
||||
`PRIVKEY_PATH`: point this to your private key you got from Certbot or similar
|
||||
`CERT_PATH`: point this to your cert you got from Certbot or similar
|
||||
|
||||
Run the server!
|
||||
|
||||
`node index.js`
|
||||
|
||||
Go to `https://whateveryourdomainis.com:3000/convert` or whatever port you selected for HTTP, and enter an RSS feed and a username.If all goes well it will create a new ActivityPub user with instructions on how to view the user.
|
||||
|
||||
## Sending out updates to followers
|
||||
|
||||
There is also a file called `updateFeeds.js` that needs to be run on a cron job or similar scheduler. I like to run mine once a minute. It queries every RSS feed in the database to see if there has been a change to the feed. If there is a new post, it sends out the new post to everyone subscribed to its corresponding ActivityPub Actor.
|
||||
|
||||
## Local testing
|
||||
|
||||
You can use a service like [ngrok](https://ngrok.com/) to test things out before you deploy on a real server. All you need to do is install ngrok and run `ngrok http 3000` (or whatever port you're using if you changed it). Then go to your `config.json` and update the `DOMAIN` field to whatever `abcdef.ngrok.io` domain that ngrok gives you and restart your server.
|
||||
|
||||
Then make sure to manually run `updateFeed.js` when the feed changes. I recommend having your own test RSS feed that you can update whenever you want.
|
||||
|
||||
## Database
|
||||
|
||||
This server uses a SQLite database to keep track of all the data. There are two tables in the database: `accounts` and `feeds`.
|
||||
|
||||
### `accounts`
|
||||
|
||||
This table keeps track of all the data needed for the accounts. Columns:
|
||||
|
||||
* `name` `TEXT PRIMARY KEY`: the account name, in the form `thename@example.com`
|
||||
* `privkey` `TEXT`: the RSA private key for the account
|
||||
* `pubkey` `TEXT`: the RSA public key for the account
|
||||
* `webfinger` `TEXT`: the entire contents of the webfinger JSON served for this account
|
||||
* `actor` `TEXT`: the entire contents of the actor JSON served for this account
|
||||
* `apikey` `TEXT`: the API key associated with this account
|
||||
* `followers` `TEXT`: a JSON-formatted array of the URL for the Actor JSON of all followers, in the form `["https://remote.server/users/somePerson", "https://another.remote.server/ourUsers/anotherPerson"]`
|
||||
* `messages` `TEXT`: not yet used but will eventually store all messages so we can render them on a "profile" page
|
||||
|
||||
### `feeds`
|
||||
|
||||
This table keeps track of all the data needed for the feeds. Columns:
|
||||
|
||||
* `feed` `TEXT PRIMARY KEY`: the URI of the RSS feed
|
||||
* `username` `TEXT`: the username associated with the RSS feed
|
||||
* `content` `TEXT`: the most recent copy fetched of the RSS feed's contents
|
||||
|
||||
## License
|
||||
|
||||
Copyright (c) 2018 Darius Kazemi. Licensed under the MIT license.
|
||||
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"DOMAIN": "",
|
||||
"PORT_HTTP": "3000",
|
||||
"PORT_HTTPS": "8443",
|
||||
"PRIVKEY_PATH": "",
|
||||
"CERT_PATH": ""
|
||||
}
|
||||
|
|
@ -0,0 +1,55 @@
|
|||
const config = require('./config.json');
|
||||
const { DOMAIN, PRIVKEY_PATH, CERT_PATH, PORT_HTTP, PORT_HTTPS } = config;
|
||||
const express = require('express');
|
||||
const app = express();
|
||||
const Database = require('better-sqlite3');
|
||||
const db = new Database('bot-node.db');
|
||||
const fs = require('fs');
|
||||
const routes = require('./routes'),
|
||||
bodyParser = require('body-parser'),
|
||||
cors = require('cors'),
|
||||
http = require('http');
|
||||
let sslOptions;
|
||||
|
||||
try {
|
||||
sslOptions = {
|
||||
key: fs.readFileSync(PRIVKEY_PATH),
|
||||
cert: fs.readFileSync(CERT_PATH)
|
||||
};
|
||||
} catch(err) {
|
||||
if (err.errno === -2) {
|
||||
console.log('No SSL key and/or cert found, not enabling https server');
|
||||
}
|
||||
else {
|
||||
console.log(err);
|
||||
}
|
||||
}
|
||||
|
||||
// if there is no `accounts` table in the DB, create an empty table
|
||||
db.prepare('CREATE TABLE IF NOT EXISTS accounts (name TEXT PRIMARY KEY, privkey TEXT, pubkey TEXT, webfinger TEXT, actor TEXT, apikey TEXT, followers TEXT, messages TEXT)').run();
|
||||
// if there is no `feeds` table in the DB, create an empty table
|
||||
db.prepare('CREATE TABLE IF NOT EXISTS feeds (feed TEXT PRIMARY KEY, username TEXT, content TEXT)').run();
|
||||
|
||||
app.set('db', db);
|
||||
app.set('domain', DOMAIN);
|
||||
app.set('port', process.env.PORT || PORT_HTTP);
|
||||
app.set('port-https', process.env.PORT_HTTPS || PORT_HTTPS);
|
||||
app.set('views', './views');
|
||||
app.set('view engine', 'pug');
|
||||
app.use(bodyParser.json({type: 'application/activity+json'})); // support json encoded bodies
|
||||
app.use(bodyParser.urlencoded({ extended: true })); // support encoded bodies
|
||||
|
||||
app.get('/', (req, res) => res.render('home'));
|
||||
|
||||
// admin page
|
||||
app.options('/api', cors());
|
||||
app.use('/api', cors(), routes.api);
|
||||
app.use('/admin', express.static('public/admin'));
|
||||
app.use('/convert', express.static('public/convert'));
|
||||
app.use('/.well-known/webfinger', cors(), routes.webfinger);
|
||||
app.use('/u', cors(), routes.user);
|
||||
app.use('/api/inbox', cors(), routes.inbox);
|
||||
|
||||
http.createServer(app).listen(app.get('port'), function(){
|
||||
console.log('Express server listening on port ' + app.get('port'));
|
||||
});
|
||||
|
|
@ -0,0 +1,26 @@
|
|||
{
|
||||
"name": "bot-node",
|
||||
"version": "1.0.0",
|
||||
"description": "",
|
||||
"main": "index.js",
|
||||
"dependencies": {
|
||||
"better-sqlite3": "^5.0.1",
|
||||
"body-parser": "^1.18.3",
|
||||
"cheerio": "^1.0.0-rc.2",
|
||||
"cors": "^2.8.4",
|
||||
"express": "^4.16.3",
|
||||
"generate-rsa-keypair": "^0.1.2",
|
||||
"pug": "^2.0.3",
|
||||
"request": "^2.87.0",
|
||||
"rss-parser": "^3.4.3"
|
||||
},
|
||||
"engines": {
|
||||
"node": ">=10.10.0"
|
||||
},
|
||||
"devDependencies": {},
|
||||
"scripts": {
|
||||
"test": "echo \"Error: no test specified\" && exit 1"
|
||||
},
|
||||
"author": "",
|
||||
"license": "MIT"
|
||||
}
|
||||
|
|
@ -0,0 +1,88 @@
|
|||
<!DOCTYPE html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="UTF-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Convert an RSS feed to ActivityPub</title>
|
||||
<style>
|
||||
body {
|
||||
font-family: sans-serif;
|
||||
max-width: 900px;
|
||||
margin: 30px;
|
||||
}
|
||||
img {
|
||||
max-width: 100px;
|
||||
}
|
||||
li {
|
||||
margin-bottom: 0.2em;
|
||||
}
|
||||
.account {
|
||||
}
|
||||
input {
|
||||
width: 300px;
|
||||
font-size: 1.2em;
|
||||
}
|
||||
.hint {
|
||||
font-size: 0.8em;
|
||||
}
|
||||
button {
|
||||
font-size: 1.2em;
|
||||
}
|
||||
</style>
|
||||
</head>
|
||||
<body>
|
||||
<h1>Convert an RSS feed to ActivityPub</h1>
|
||||
<p>Put the full RSS feed URL in here, and pick a username for the account that will track the feed.</p>
|
||||
<p>
|
||||
<input id="feed" type="text" placeholder="https://example.com/feed.xml"/>
|
||||
</p>
|
||||
<p>
|
||||
<input id="username" type="text" placeholder="username"/><br><span class="hint">only letters, digits, and underscore (_) allowed</span>
|
||||
</p>
|
||||
<button onclick="submit()">Submit</button>
|
||||
<div id="out">
|
||||
</div>
|
||||
|
||||
<script>
|
||||
// https://bots.tinysubversions.com/api/convert/?feed=https://toomuchnotenough.site/feed.xml&username=tmne
|
||||
function submit() {
|
||||
let feed = document.querySelector('#feed').value;
|
||||
let username = document.querySelector('#username').value;
|
||||
let out = document.querySelector('#out');
|
||||
fetch(`/api/convert/?feed=${feed}&username=${username}`)
|
||||
.then(function(response) {
|
||||
if (response.status !== 200) {
|
||||
out.innerHTML = `<p>Error: ${JSON.stringify(response.statusText)}</p>`;
|
||||
return {};
|
||||
}
|
||||
return response.json();
|
||||
})
|
||||
.then(function(myJson) {
|
||||
console.log((myJson));
|
||||
// a feed exists in the database
|
||||
if (myJson.content) {
|
||||
// was it a match on feed
|
||||
if (myJson.feed === feed) {
|
||||
console.log('feed match!');
|
||||
out.innerHTML = `<p>This feed already exists! Follow @${myJson.username}@bots.tinysubversions.com.</p>`;
|
||||
window.location = `/u/${myJson.username}`;
|
||||
}
|
||||
// was it a match on username
|
||||
else if (myJson.username === username) {
|
||||
console.log('username match!');
|
||||
out.innerHTML = `<p>This username is already taken for <a href="${myJson.feed}">this feed</a>.</p>`;
|
||||
}
|
||||
}
|
||||
else if (myJson.title) {
|
||||
out.innerHTML = `<p>Okay! There is now an ActivityPub actor for ${myJson.title}. You should be able to search for it from your ActivityPub client (Mastodon, Pleroma, etc) using this identifier: @${username}@bots.tinysubversions.com. You won't see anything there until the next time the RSS feed updates. You can check out the profile page for this feed at <a href="https://bots.tinysubversions.com/u/${username}/">https://bots.tinysubversions.com/u/${username}</a> too!</p>`;
|
||||
}
|
||||
|
||||
})
|
||||
.catch(error => {
|
||||
console.log('!!!',error);
|
||||
out.innerHTML = `<p>Error: ${error}</p>`;
|
||||
});
|
||||
}
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
|
|
@ -0,0 +1,105 @@
|
|||
'use strict';
|
||||
const express = require('express'),
|
||||
router = express.Router(),
|
||||
crypto = require('crypto'),
|
||||
Parser = require('rss-parser'),
|
||||
generateRSAKeypair = require('generate-rsa-keypair');
|
||||
|
||||
router.get('/convert', function (req, res) {
|
||||
let db = req.app.get('db');
|
||||
console.log(req.query);
|
||||
let username = req.query.username;
|
||||
let feed = req.query.feed;
|
||||
// reject if username is invalid
|
||||
if (username.match(/^[a-zA-Z0-9_]+$/) === null) {
|
||||
return res.status(400).json('Invalid username! Only alphanumerics and underscore (_) allowed.');
|
||||
}
|
||||
// check to see if feed exists
|
||||
let result = db.prepare('select * from feeds where feed = ? or username = ?').get(feed, username);
|
||||
// see if we already have an entry for this feed
|
||||
if (result) {
|
||||
// return feed
|
||||
res.status(200).json(result);
|
||||
}
|
||||
else if(feed && username) {
|
||||
console.log('VALIDATING');
|
||||
// validate the RSS
|
||||
let parser = new Parser();
|
||||
parser.parseURL(feed, function(err, feedData) {
|
||||
if (err) {
|
||||
res.status(400).json({err: err.message});
|
||||
}
|
||||
else {
|
||||
console.log(feedData.title);
|
||||
console.log('end!!!!');
|
||||
res.status(200).json(feedData);
|
||||
let displayName = feedData.title;
|
||||
let account = username;
|
||||
// create new user
|
||||
let db = req.app.get('db');
|
||||
let domain = req.app.get('domain');
|
||||
// create keypair
|
||||
var pair = generateRSAKeypair();
|
||||
let imageUrl = null;
|
||||
// if image exists set image
|
||||
if (feedData.image && feedData.image.url) {
|
||||
imageUrl = feedData.image.url;
|
||||
}
|
||||
let actorRecord = createActor(account, domain, pair.public, displayName, imageUrl);
|
||||
let webfingerRecord = createWebfinger(account, domain);
|
||||
const apikey = crypto.randomBytes(16).toString('hex');
|
||||
db.prepare('insert or replace into accounts(name, actor, apikey, pubkey, privkey, webfinger) values(?, ?, ?, ?, ?, ?)').run( `${account}@${domain}`, apikey, pair.public, pair.private, JSON.stringify(actorRecord), JSON.stringify(webfingerRecord));
|
||||
let content = JSON.stringify(feedData);
|
||||
db.prepare('insert or replace into feeds(feed, username, content) values(?, ?, ?)').run( feed, username, content);
|
||||
}
|
||||
});
|
||||
}
|
||||
else {
|
||||
res.status(404).json({msg: 'unknown error'});
|
||||
}
|
||||
});
|
||||
|
||||
function createActor(name, domain, pubkey, displayName, imageUrl) {
|
||||
displayName = displayName || name;
|
||||
let actor = {
|
||||
'@context': [
|
||||
'https://www.w3.org/ns/activitystreams',
|
||||
'https://w3id.org/security/v1'
|
||||
],
|
||||
|
||||
'id': `https://${domain}/u/${name}`,
|
||||
'type': 'Person',
|
||||
'preferredUsername': `${name}`,
|
||||
'inbox': `https://${domain}/api/inbox`,
|
||||
'name': displayName,
|
||||
'publicKey': {
|
||||
'id': `https://${domain}/u/${name}#main-key`,
|
||||
'owner': `https://${domain}/u/${name}`,
|
||||
'publicKeyPem': pubkey
|
||||
}
|
||||
};
|
||||
if (imageUrl) {
|
||||
actor.icon = {
|
||||
'type': 'Image',
|
||||
'mediaType': 'image/png',
|
||||
'url': imageUrl,
|
||||
};
|
||||
}
|
||||
return actor;
|
||||
}
|
||||
|
||||
function createWebfinger(name, domain) {
|
||||
return {
|
||||
'subject': `acct:${name}@${domain}`,
|
||||
|
||||
'links': [
|
||||
{
|
||||
'rel': 'self',
|
||||
'type': 'application/activity+json',
|
||||
'href': `https://${domain}/u/${name}`
|
||||
}
|
||||
]
|
||||
};
|
||||
}
|
||||
|
||||
module.exports = router;
|
||||
|
|
@ -0,0 +1,95 @@
|
|||
'use strict';
|
||||
const express = require('express'),
|
||||
crypto = require('crypto'),
|
||||
request = require('request'),
|
||||
fs = require('fs'),
|
||||
router = express.Router();
|
||||
|
||||
function signAndSend(message, name, domain, req, res, targetDomain) {
|
||||
// get the private key
|
||||
let db = req.app.get('db');
|
||||
let result = db.prepare('select privkey from accounts where name = ?').get(`${name}@${domain}`);
|
||||
if (result === undefined) {
|
||||
return res.status(404).send(`No record found for ${name}.`);
|
||||
}
|
||||
else {
|
||||
let privkey = result.privkey;
|
||||
const signer = crypto.createSign('sha256');
|
||||
let d = new Date();
|
||||
let stringToSign = `(request-target): post /inbox\nhost: ${targetDomain}\ndate: ${d.toUTCString()}`;
|
||||
signer.update(stringToSign);
|
||||
signer.end();
|
||||
const signature = signer.sign(privkey);
|
||||
const signature_b64 = signature.toString('base64');
|
||||
let header = `keyId="https://${domain}/u/${name}",headers="(request-target) host date",signature="${signature_b64}"`;
|
||||
console.log('signature:',header);
|
||||
request({
|
||||
url: `https://${targetDomain}/inbox`,
|
||||
headers: {
|
||||
'Host': targetDomain,
|
||||
'Date': d.toUTCString(),
|
||||
'Signature': header
|
||||
},
|
||||
method: 'POST',
|
||||
json: true,
|
||||
body: message
|
||||
}, function (error, response, body){
|
||||
});
|
||||
res.json('done');
|
||||
}
|
||||
}
|
||||
|
||||
function sendAcceptMessage(thebody, name, domain, req, res, targetDomain) {
|
||||
const guid = crypto.randomBytes(16).toString('hex');
|
||||
console.log(thebody);
|
||||
let message = {
|
||||
'@context': 'https://www.w3.org/ns/activitystreams',
|
||||
'id': `https://${domain}/${guid}`,
|
||||
'type': 'Accept',
|
||||
'actor': `https://${domain}/u/${name}`,
|
||||
'object': thebody,
|
||||
};
|
||||
signAndSend(message, name, domain, req, res, targetDomain);
|
||||
}
|
||||
|
||||
router.post('/', function (req, res) {
|
||||
// pass in a name for an account, if the account doesn't exist, create it!
|
||||
let domain = req.app.get('domain');
|
||||
const myURL = new URL(req.body.actor);
|
||||
let targetDomain = myURL.hostname;
|
||||
fs.appendFile('/home/dariusk/bot-node/inbox.log', JSON.stringify(req.body)+'\r\n', function (err) {
|
||||
if (err) {
|
||||
return console.log(err);
|
||||
}
|
||||
});
|
||||
// TODO: add "Undo" follow event
|
||||
if (typeof req.body.object === 'string' && req.body.type === 'Follow') {
|
||||
let name = req.body.object.replace(`https://${domain}/u/`,'');
|
||||
sendAcceptMessage(req.body, name, domain, req, res, targetDomain);
|
||||
// Add the user to the DB of accounts that follow the account
|
||||
let db = req.app.get('db');
|
||||
// get the followers JSON for the user
|
||||
let result = db.prepare('select followers from accounts where name = ?').get(`${name}@${domain}`);
|
||||
if (result === undefined) {
|
||||
console.log(`No record found for ${name}.`);
|
||||
}
|
||||
else {
|
||||
// update followers
|
||||
let followers = result.followers;
|
||||
console.log(followers);
|
||||
if (followers) {
|
||||
followers.push(req.body.actor);
|
||||
// unique items
|
||||
followers = [...new Set(followers)];
|
||||
}
|
||||
else {
|
||||
followers = [req.body.actor];
|
||||
}
|
||||
let followersText = JSON.stringify(followers);
|
||||
// update into DB
|
||||
db.prepare('update accounts set followers = ? where name = ?').run(`${name}@${domain}`, followersText);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
|
|
@ -0,0 +1,8 @@
|
|||
'use strict';
|
||||
|
||||
module.exports = {
|
||||
api: require('./api'),
|
||||
user: require('./user'),
|
||||
inbox: require('./inbox'),
|
||||
webfinger: require('./webfinger'),
|
||||
};
|
||||
|
|
@ -0,0 +1,40 @@
|
|||
'use strict';
|
||||
const express = require('express'),
|
||||
router = express.Router();
|
||||
|
||||
router.get('/:name', function (req, res) {
|
||||
let name = req.params.name;
|
||||
if (!name) {
|
||||
return res.status(400).send('Bad request.');
|
||||
}
|
||||
else {
|
||||
let db = req.app.get('db');
|
||||
let domain = req.app.get('domain');
|
||||
name = `${name}@${domain}`;
|
||||
let result = db.prepare('select actor from accounts where name = ?').get(name);
|
||||
if (result === undefined) {
|
||||
return res.status(404).json(`No record found for ${name}.`);
|
||||
}
|
||||
else if (req.headers.accept.includes('application/activity+json') || req.headers.accept.includes('application/json') || req.headers.accept.includes('application/json+ld')) {
|
||||
res.json(result.actor);
|
||||
}
|
||||
else {
|
||||
let actor = result.actor;
|
||||
let username = name.replace('@'+domain,'');
|
||||
console.log(username, actor);
|
||||
let content = db.prepare('select content from feeds where username = ?').get(username);
|
||||
if (content === undefined) {
|
||||
return res.status(404).json(`Something went very wrong!`);
|
||||
}
|
||||
let feedData = JSON.parse(content.content);
|
||||
let imageUrl = null;
|
||||
// if image exists set image
|
||||
if (feedData.image && feedData.image.url) {
|
||||
imageUrl = feedData.image.url;
|
||||
}
|
||||
res.render('user', { displayName: actor.name, items: feedData.items, accountName: '@'+name, imageUrl: imageUrl });
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
'use strict';
|
||||
const express = require('express'),
|
||||
router = express.Router();
|
||||
|
||||
router.get('/', function (req, res) {
|
||||
let resource = req.query.resource;
|
||||
if (!resource || !resource.includes('acct:')) {
|
||||
return res.status(400).send('Bad request. Please make sure "acct:USER@DOMAIN" is what you are sending as the "resource" query parameter.');
|
||||
}
|
||||
else {
|
||||
let name = resource.replace('acct:','');
|
||||
let db = req.app.get('db');
|
||||
let result = db.prepare('select webfinger from accounts where name = ?').get(name);
|
||||
if (result === undefined) {
|
||||
return res.status(404).send(`No record found for ${name}.`);
|
||||
}
|
||||
else {
|
||||
res.json(result.webfinger);
|
||||
}
|
||||
}
|
||||
});
|
||||
|
||||
module.exports = router;
|
||||
|
|
@ -0,0 +1,197 @@
|
|||
const Database = require('better-sqlite3');
|
||||
const db = new Database('bot-node.db'),
|
||||
Parser = require('rss-parser'),
|
||||
request = require('request'),
|
||||
crypto = require('crypto'),
|
||||
parser = new Parser();
|
||||
|
||||
// get all feeds from DB
|
||||
let feeds = db.prepare('select * from feeds').all();
|
||||
for (var feed of feeds) {
|
||||
// fetch new RSS for each feed
|
||||
parser.parseURL(feed.feed, function(err, feedData) {
|
||||
if (err) {
|
||||
console.log('error fetching', feed.feed, err);
|
||||
}
|
||||
else {
|
||||
//console.log(feedData);
|
||||
// get the old feed data from the database
|
||||
let oldFeed = JSON.parse(feed.content);
|
||||
|
||||
// compare the feed item contents. if there's one or more whole new items (aka a new item with a unique guid),
|
||||
// add the items to a list like
|
||||
// [ { items: [], username }, {}, ... ]
|
||||
|
||||
let oldItems = oldFeed.items;
|
||||
let newItems = feedData.items;
|
||||
|
||||
// find the difference of the sets of guids (fall back to title or
|
||||
// description since guid is not required by spec) in the old and new feeds
|
||||
|
||||
let oldGuidSet = new Set(oldItems.map(el => el.guid || el.title || el.description));
|
||||
let newGuidSet = new Set(newItems.map(el => el.guid || el.title || el.description));
|
||||
// find things in the new set that aren't in the old set
|
||||
let difference = new Set( [...newGuidSet].filter(x => !oldGuidSet.has(x)));
|
||||
difference = [...difference];
|
||||
|
||||
console.log('diff', difference);
|
||||
|
||||
if (difference.length > 0) {
|
||||
// get a list of new items in the diff
|
||||
let brandNewItems = newItems.filter(el => difference.includes(el.guid) || difference.includes(el.title) || difference.includes(el.description));
|
||||
let acct = feed.username;
|
||||
let domain = 'bots.tinysubversions.com';
|
||||
console.log(acct, brandNewItems);
|
||||
|
||||
// send the message to everyone for each item!
|
||||
for (var item of brandNewItems) {
|
||||
// FIX THIS
|
||||
item = transformContent(item);
|
||||
console.log(item.urls);
|
||||
let message = `<p><a href="${item.link}">${item.title}</a></p><p>${item.content}</p>`;
|
||||
if (item.enclosure && item.enclosure.url && item.enclosure.url.includes('.mp3')) {
|
||||
message += `<p><a href="${item.enclosure.url}">${item.enclosure.url}</a></p>`;
|
||||
}
|
||||
sendCreateMessage(message, acct, domain, null, null, item);
|
||||
}
|
||||
|
||||
// update the DB with new contents
|
||||
let content = JSON.stringify(feedData);
|
||||
db.prepare('insert or replace into feeds(feed, username, content) values(?, ?, ?)').run( feed, acct, content);
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
// TODO: update the display name of a feed if the feed title has changed
|
||||
|
||||
// This is a function with a bunch of custom rules for different kinds of content I've found in the wild in things like Reddit rss feeds
|
||||
function transformContent(item) {
|
||||
let cheerio = require('cheerio');
|
||||
console.log(item.content);
|
||||
if (item.content === undefined) {
|
||||
item.urls = [];
|
||||
return item;
|
||||
}
|
||||
let $ = cheerio.load(item.content);
|
||||
|
||||
// look through all the links
|
||||
let links = $('a');
|
||||
let urls = [];
|
||||
console.log('links', links.length);
|
||||
links.each((i,e) => {
|
||||
let url = $(e).attr('href');
|
||||
// if there's an image, add it as a media attachment
|
||||
if (url.match(/(http)?s?:?(\/\/[^"']*\.(?:png|jpg|jpeg|gif|png|svg))/)) {
|
||||
console.log(url);
|
||||
urls.push(url);
|
||||
}
|
||||
});
|
||||
item.urls = urls;
|
||||
|
||||
// remove multiple line breaks
|
||||
$('br+br+br').remove();
|
||||
$('p').each((i, el) => {
|
||||
if($(el).html().replace(/\s| /g, '').length === 0) {$(el).remove();}
|
||||
});
|
||||
|
||||
// convert li items to bullet points
|
||||
$('li').each((i, el) => {
|
||||
console.log($(el).html());
|
||||
$(el).replaceWith(`<span>- ${$(el).html()}</span><br>`);
|
||||
});
|
||||
|
||||
item.content = $('body').html();
|
||||
return item;
|
||||
}
|
||||
|
||||
// for each item in the list, get the account corresponding to the username
|
||||
// for each item in the ITEMS list, send a message to all followers
|
||||
|
||||
// TODO import these form a helper
|
||||
function signAndSend(message, name, domain, req, res, targetDomain, inbox) {
|
||||
// get the private key
|
||||
console.log('sending to ', name, targetDomain, inbox);
|
||||
let inboxFragment = inbox.replace('https://'+targetDomain,'');
|
||||
let result = db.prepare('select privkey from accounts where name = ?').get(name);
|
||||
console.log('got key', result === undefined, `${name}@${domain}`);
|
||||
if (result === undefined) {
|
||||
console.log(`No record found for ${name}.`);
|
||||
}
|
||||
else {
|
||||
let privkey = result.privkey;
|
||||
const signer = crypto.createSign('sha256');
|
||||
let d = new Date();
|
||||
let stringToSign = `(request-target): post ${inboxFragment}\nhost: ${targetDomain}\ndate: ${d.toUTCString()}`;
|
||||
signer.update(stringToSign);
|
||||
signer.end();
|
||||
const signature = signer.sign(privkey);
|
||||
const signature_b64 = signature.toString('base64');
|
||||
let header = `keyId="https://${domain}/u/${name}",headers="(request-target) host date",signature="${signature_b64}"`;
|
||||
console.log('signature:',header);
|
||||
request({
|
||||
url: inbox,
|
||||
headers: {
|
||||
'Host': targetDomain,
|
||||
'Date': d.toUTCString(),
|
||||
'Signature': header
|
||||
},
|
||||
method: 'POST',
|
||||
json: true,
|
||||
body: message
|
||||
}, function (error, response, body){
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
function createMessage(text, name, domain, item) {
|
||||
const guid = crypto.randomBytes(16).toString('hex');
|
||||
let d = new Date();
|
||||
|
||||
let out = {
|
||||
'@context': 'https://www.w3.org/ns/activitystreams',
|
||||
|
||||
'id': `https://${domain}/${guid}`,
|
||||
'type': 'Create',
|
||||
'actor': `https://${domain}/u/${name}`,
|
||||
|
||||
'object': {
|
||||
'id': `https://${domain}/${guid}`,
|
||||
'type': 'Note',
|
||||
'published': d.toISOString(),
|
||||
'attributedTo': `https://${domain}/u/${name}`,
|
||||
'content': text,
|
||||
'to': 'https://www.w3.org/ns/activitystreams#Public'
|
||||
}
|
||||
};
|
||||
|
||||
// add image attachment
|
||||
let attachment;
|
||||
if (item.urls.length > 0) {
|
||||
console.log('appending');
|
||||
attachment = {
|
||||
'type': 'Document',
|
||||
'mediaType': 'image/png', // TODO: update the mediaType to match jpeg,gif,etc
|
||||
'url': item.urls[0],
|
||||
'name': null
|
||||
};
|
||||
out.object.attachment = attachment;
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
function sendCreateMessage(text, name, domain, req, res, item) {
|
||||
let message = createMessage(text, name, domain, item);
|
||||
|
||||
let result = db.prepare('select followers from accounts where name = ?').get(`${name}@${domain}`);
|
||||
let followers = JSON.parse(result.followers);
|
||||
console.log(followers);
|
||||
for (let follower of followers) {
|
||||
let inbox = follower+'/inbox';
|
||||
let myURL = new URL(follower);
|
||||
let targetDomain = myURL.hostname;
|
||||
signAndSend(message, name, domain, req, res, targetDomain, inbox);
|
||||
}
|
||||
}
|
||||
|
||||
|
|
@ -0,0 +1,11 @@
|
|||
html
|
||||
head
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
title Feed Converter
|
||||
style
|
||||
include style.css
|
||||
body
|
||||
h1 RSS to ActivityPub Converter
|
||||
p.account by <a href="https://friend.camp/@darius">Darius Kazemi</a>
|
||||
p This is a service to convert any RSS feed into an account that Mastodon (or any other ActivityPub social network) can subscribe to.
|
||||
p <a href="/convert">Click here to start!</a>
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
body {
|
||||
font-family: sans-serif;
|
||||
max-width: 900px;
|
||||
margin: 30px;
|
||||
}
|
||||
img {
|
||||
max-width: 100px;
|
||||
}
|
||||
li {
|
||||
margin-bottom: 0.2em;
|
||||
}
|
||||
.account {
|
||||
}
|
||||
input {
|
||||
width: 300px;
|
||||
font-size: 1.2em;
|
||||
}
|
||||
.hint {
|
||||
font-size: 0.8em;
|
||||
}
|
||||
button {
|
||||
font-size: 1.2em;
|
||||
}
|
||||
|
|
@ -0,0 +1,15 @@
|
|||
html
|
||||
head
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
title= displayName
|
||||
style
|
||||
include style.css
|
||||
body
|
||||
h1= displayName
|
||||
img(src=imageUrl)
|
||||
p.account #{accountName}
|
||||
p What you're looking at is an RSS feed that's been converted into an account that Mastodon (or any other ActivityPub social network) can subscribe to. Put the username above into your user search and you should be able to find this feed and subscribe!
|
||||
h2 Feed items
|
||||
ul
|
||||
for item in items
|
||||
li: a(href=item.link) #{item.title}
|
||||
Loading…
Reference in New Issue