Require OAuth 2.0 verification to create feeds

As of the `v2.0.0` release of this project, only users who are authenticated with a particular OAuth server can _create_ feeds. Any federated user can still read the feeds. I implemented this because running this service in the open invited thousands of spammers to create feeds and overwhelm the service. With this new model, you can run this as an added bonus for people in a community like a Mastodon server, and as the person running it you are taking on only the moderation burden of the users you are already responsible for on your federated server.

README.md

@@ -4,12 +4,16 @@ This is a server that lets users convert any RSS feed to an ActivityPub actor th
 
 This is based on my [Express ActivityPub Server](https://github.com/dariusk/express-activitypub), a simple Node/Express server that supports a subset of ActivityPub.
 
+As of the `v2.0.0` release of this project, only users who are authenticated with a particular OAuth server can _create_ feeds. Any federated user can still read the feeds. I implemented this because running this service in the open invited thousands of spammers to create feeds and overwhelm the service. With this new model, you can run this as an added bonus for people in a community like a Mastodon server, and as the person running it you are taking on only the moderation burden of the users you are already responsible for on your federated server.
+
 ## Requirements
 
 This requires Node.js v10.10.0 or above.
 
 You also need `beanstalkd` running. This is a simple and fast queueing system we use to manage polling RSS feeds. [Here are installation instructions](https://beanstalkd.github.io/download.html). On a production server you'll want to [install it as a background process](https://github.com/beanstalkd/beanstalkd/tree/master/adm).
 
+You'll also need to control some kind of OAuth provider that you can regsiter this application on. This application was designed to work with Mastodon as that OAuth provider (see more on setting that up below), but any OAuth 2.0 provider should work. Many federated software packages besides Mastodon can act as OAuth providers, and if you want something standalone, [Keycloak](https://www.keycloak.org) and [ORY Hydra](https://github.com/ory/hydra) are two open source providers you could try.
+
 ## Installation
 
 Clone the repository, then `cd` into its root directory. Install dependencies:
@@ -28,7 +32,17 @@ Update your new `config.json` file:
   "PORT_HTTP": "3000",
   "PORT_HTTPS": "8443",
   "PRIVKEY_PATH": "/path/to/your/ssl/privkey.pem",
-  "CERT_PATH": "/path/to/your/ssl/cert.pem"
+  "CERT_PATH": "/path/to/your/ssl/cert.pem",
+  "OAUTH": {
+    "client_id": "abc123def456",
+    "client_secret": "zyx987wvu654",
+    "redirect_uri": "https://rss.example.social/convert",
+    "domain": "example.social",
+    "domain_human": "Example Online Community",
+    "authorize_path": "/oauth/authorize",
+    "token_path": "/oauth/token",
+    "token_verification_path": "/some/path/to/verify/token"
+  }
 }
 ```
 
@@ -37,6 +51,15 @@ Update your new `config.json` file:
 * `PORT_HTTPS`: the https port that Express runs on
 * `PRIVKEY_PATH`: point this to your private key you got from Certbot or similar
 * `CERT_PATH`: point this to your cert you got from Certbot or similar
+* `OAUTH`: this object contains properties related to OAuth login. See the section below on "Running with OAuth" for more details.
+  * `client_id`: also known as the "client key". A long series of characters. You generate this when you register this application with an OAuth provider.
+  * `client_secret`: Another long series of characters that you generate when you register this application with an OAuth provider.
+  * `redirect_uri`: This is the URI that people get redirected to after they authorize the application on the OAuth server. Must point to the server where THIS service is running, and must point to the `/convert` page. This uri has to match what you put in the application info on the OAuth provider.
+  * `domain`: The domain of the OAuth provider. Not necessarily the same as this server (for example, you could host this at rss.mydomain.com and then handle all OAuth through some other server you control, like a Mastodon server).
+  * `domain_human`: The human-readable name of the OAuth provider. This will appear in various messages, so if you say "Example Online Community" here then the user will see a message like "Click here to log in via Example Online Community".
+  * `authorize_path`: This will generally be `/oauth/authorize/` but you can change it here if your OAuth provider uses a nonstandard authorization path.
+  * `token_path`: This will generally be `/oauth/token/` but you can change it here if your OAuth provider uses a nonstandard token path.
+  * `token_verification_path`: This should be the path to any URL at the OAuth server that responds with an HTTP status code 200 when you are correctly logged in (and with a non-200 value when you are not). This is the path relative to the `domain` you set, so if your `domain` is `example.social` and you set `token_verification_path` to `/foo/bar/` then the full path that this service will run a GET on to verify you are logged in is `https://example.social/foo/bar`.
 
 Run the server!
 
@@ -48,6 +71,27 @@ Go to `https://whateveryourdomainis.com:3000/convert` or whatever port you selec
 
 There is also a file called `queueFeeds.js` that needs to be run on a cron job or similar scheduler. I like to run mine once a minute. It queries every RSS feed in the database to see if there has been a change to the feed. If there is a new post, it sends out the new post to everyone subscribed to its corresponding ActivityPub Actor.
 
+## Running with OAuth
+
+OAuth is unfortunately a bit underspecified so there are a lot of funky implementations out there. Here I will include an example of using a Mastodon server as the OAuth provider. This is how I have my RSS service set up: I run friend.camp as my Mastodon server, and I use my admin powers on friend.camp to register rss.friend.camp as an application. The steps for this, for Mastodon, are:
+
+* log in as an admin user
+* go to Preferences
+* select Development
+* select New Application
+* type in an application name, and the URL where this service is running
+* type in the redirect URI, which will be whatever base domain this service is running at with the `/convert` path appended. So something like `https://rss.example.social/convert`
+* uncheck all scopes, and check `read:accounts` (this is the minimum required access, simply so this RSS converter can confirm someone is truly logged in)
+* once you're done, save
+* you will now have access to a "client key" and "client secret" for this app.
+* open `config.js` in an editor
+* fill in `client_id` with the client key, and `client_secret` with the client secret.
+* set the `redirect_uri` to be identical to the one you put in Mastodon. It should look like `https://rss.example.social/convert` (the `/convert` part is important, this software won't work if you point to a different path)
+* set `domain` to the domain of your Mastodon server, and `domain_human` to its human-friendly name
+* leave `authorize_path` and `token_path` on their defaults
+* set `token_verification_path` to `/api/v1/accounts/verify_credentials`
+* cross your fingers and start up this server
+
 ## Local testing
 
 You can use a service like [ngrok](https://ngrok.com/) to test things out before you deploy on a real server. All you need to do is install ngrok and run `ngrok http 3000` (or whatever port you're using if you changed it). Then go to your `config.json` and update the `DOMAIN` field to whatever `abcdef.ngrok.io` domain that ngrok gives you and restart your server.

index.js

@@ -1,5 +1,5 @@
 const config = require('./config.json');
-const { DOMAIN, PRIVKEY_PATH, CERT_PATH, PORT_HTTP, PORT_HTTPS } = config;
+const { DOMAIN, PRIVKEY_PATH, CERT_PATH, PORT_HTTP, PORT_HTTPS, OAUTH } = config;
 const express = require('express');
 const app = express();
 const Database = require('better-sqlite3');
@@ -41,7 +41,7 @@ app.set('view engine', 'pug');
 app.use(bodyParser.json({type: 'application/activity+json'})); // support json encoded bodies
 app.use(bodyParser.urlencoded({ extended: true })); // support encoded bodies
 
-app.get('/', (req, res) => res.render('home'));
+app.get('/', (req, res) => res.render('home', { OAUTH }));
 
 // admin page
 app.options('/api', cors());

package.json

@@ -1,6 +1,6 @@
 {
-  "name": "bot-node",
+  "name": "rss-to-activitypub",
-  "version": "1.0.0",
+  "version": "2.0.0",
   "description": "",
   "main": "index.js",
   "dependencies": {

public/convert/index.html

@@ -33,6 +33,8 @@
 <body>
 <h1>Convert an RSS feed to ActivityPub</h1>
 <p><em>by <a href="https://friend.camp/@darius">Darius Kazemi</a>, <a href="https://github.com/dariusk/rss-to-activitypub">source code here</a></em></p>
+<div id="convert">
+<p id="login-confirmed"></p>
 <p>Put the full RSS feed URL in here, and pick a username for the account that will track the feed.</p>
 <p>
 <input id="feed" type="text" placeholder="https://example.com/feed.xml"/>
@@ -44,15 +46,18 @@
 <button onclick="submit()">Submit</button>
 <div id="out">
 </div>
+</div>
+<div id="login" hidden=true>
+<p>You aren't logged in! <a href="/">Go here to log in.</a></p>
+</div>
 
 <script>
-// https://bots.tinysubversions.com/api/convert/?feed=https://toomuchnotenough.site/feed.xml&username=tmne
 function submit() {
 let domain = document.domain;
 let feed = encodeURIComponent(document.querySelector('#feed').value);
 let username = document.querySelector('#username').value;
 let out = document.querySelector('#out');
-fetch(`/api/convert/?feed=${feed}&username=${username}`)
+fetch(`/api/convert/?feed=${feed}&username=${username}&token=${access_token}`)
   .then(function(response) {
     return response.json();
   })
@@ -65,13 +70,11 @@ fetch(`/api/convert/?feed=${feed}&username=${username}`)
     if (myJson.content) {
       // was it a match on feed
       if (myJson.feed === decodeURIComponent(feed)) {
-        console.log('feed match!');
         out.innerHTML = `<p>This feed already exists! Follow @${myJson.username}@${domain}.</p>`;
         window.location = `/u/${myJson.username}`;
       }
       // was it a match on username
       else if (myJson.username === username) {
-        console.log('username match!');
         out.innerHTML = `<p>This username is already taken for <a href="${myJson.feed}">this feed</a>.</p>`;
       }
     }
@@ -81,10 +84,53 @@ fetch(`/api/convert/?feed=${feed}&username=${username}`)
 
   })
   .catch(error => {
-    console.log('!!!',error);
     out.innerHTML = `<p>Error: ${error}</p>`;
   });
 }
+
+function getUrlParameter(name) {
+  name = name.replace(/[\[]/, '\\[').replace(/[\]]/, '\\]');
+  var regex = new RegExp('[\\?&]' + name + '=([^&#]*)');
+  var results = regex.exec(location.search);
+  return results === null ? '' : decodeURIComponent(results[1].replace(/\+/g, ' '));
+};
+
+if (!localStorage.getItem('rss-data')) {
+  localStorage.setItem('rss-data','{}');
+}
+let {access_token, domain} = JSON.parse(localStorage.getItem('rss-data'));
+
+// if no access token in storage, no code in url
+// hide app, show login prompt
+if (!getUrlParameter('code') && !access_token) {
+  document.getElementById('convert').hidden = true;
+  document.getElementById('login').hidden = false;
+}
+// if no access token in storage, code in url
+// send the code parameter to the server to get an access token
+// store the result in localStorage, reload the page
+if (getUrlParameter('code') && !access_token) {
+  fetch(`/api/request-token/?code=${getUrlParameter('code')}`)
+    .then((resp) => resp.json())
+    .then(data => {
+      localStorage.setItem('rss-data',JSON.stringify(data));
+      // reload without url parameters
+      window.location.href = window.location.origin + window.location.pathname;
+    });
+}
+// if we have an access token, then we can render our app
+if (JSON.parse(localStorage.getItem('rss-data')).access_token) {
+  let {access_token, domain} = JSON.parse(localStorage.getItem('rss-data'));
+
+  document.getElementById('login').hidden = true;
+  document.getElementById('login-confirmed').innerHTML = `<p>Welcome! You are logged in via your <strong>${domain}</strong> account. <a href="#" onclick="logOut()">Click here</a> to log out.</p>`;
+  document.getElementById('convert').hidden = false;
+}
+
+function logOut() {
+	localStorage.removeItem('rss-data');
+	window.location = window.location;
+}
 </script>
 </body>
 </html>

queueFeeds.js

@@ -9,7 +9,7 @@ async function foo() {
   // get all feeds from DB
   let feeds = db.prepare('select feed from feeds').all();
 
-  console.log('!!!',feeds.length);
+  // console.log('!!!',feeds.length);
 
   let count = 0;
 

routes/api.js

@@ -1,15 +1,66 @@
 'use strict';
 const express = require('express'),
       router = express.Router(),
+      cors = require('cors'),
       crypto = require('crypto'),
       request = require('request'),
       Parser = require('rss-parser'),
       parseFavicon = require('parse-favicon').parseFavicon,
-      generateRSAKeypair = require('generate-rsa-keypair');
+      generateRSAKeypair = require('generate-rsa-keypair'),
+      oauth = require('../config.json').OAUTH;
 
-router.get('/convert', function (req, res) {
+router.get('/request-token', cors(), (req, res) => {
+  if (!oauth) {
+    return res.status(501).json({message: `OAuth is not enabled on this server.`});
+  }
+  else if (!oauth.client_id || !oauth.client_secret || !oauth.redirect_uri) {
+    return res.status(501).json({message: `OAuth is misconfigured on this server. Please contact the admin at ${contactEmail} and let them know.`});
+  }
+  else if (!req.query.code) {
+    return res.status(400).json({message: `Request is missing the required 'code' parameter.`});
+  }
+
+  let params = req.query;
+  params.client_id = oauth.client_id;
+  params.client_secret = oauth.client_secret;
+  params.redirect_uri = oauth.redirect_uri;
+  params.grant_type = 'authorization_code';
+  request.post(`https://${oauth.domain}${oauth.token_path}`, {form: params}, (err,httpResponse,body) => {
+    body = JSON.parse(body);
+    if (body.access_token) {
+      return res.json({ access_token: body.access_token, domain: oauth.domain});
+    }
+    else {
+      return res.status(401).json(body);
+    }
+  });
+});
+
+// if oauth is enabled, this function checks to see if we've been sent an access token and validates it with the server
+// otherwise we simply skip verification
+function isAuthenticated(req, res, next) {
+  if (oauth) {
+    request.get({
+      url: `https://${oauth.domain}${oauth.token_verification_path}`,
+      headers: {
+        'Authorization': `Bearer ${req.query.token}`
+      },
+    }, (err, resp, body) => {
+      if (resp.statusCode === 200) {
+        return next();
+      }
+      else {
+        res.redirect('/');
+      }
+    });
+  }
+  else {
+    return next();
+  }
+}
+
+router.get('/convert', isAuthenticated, function (req, res) {
   let db = req.app.get('db');
-  console.log(req.query);
   let username = req.query.username;
   let feed = req.query.feed;
   // reject if username is invalid
@@ -24,7 +75,6 @@ router.get('/convert', function (req, res) {
     res.status(200).json(result);
   }
   else if(feed && username) {
-    console.log('VALIDATING');
     // validate the RSS
     let parser = new Parser();
     parser.parseURL(feed, function(err, feedData) {
@@ -35,8 +85,6 @@ router.get('/convert', function (req, res) {
         res.status(400).json({err: err.message});
       }
       else {
-        console.log(feedData.title);
-        console.log('end!!!!');
         res.status(200).json(feedData);
         let displayName = feedData.title;
         let description = feedData.description;

routes/inbox.js

@@ -29,9 +29,6 @@ function signAndSend(message, name, domain, req, res, targetDomain) {
     const signature_b64 = signature.toString('base64');
     const algorithm = 'rsa-sha256';
     let header = `keyId="https://${domain}/u/${name}",algorithm="${algorithm}",headers="(request-target) host date digest",signature="${signature_b64}"`;
-    console.log('signature:',header);
-    console.log('message:',message);
-
     request({
       url: inbox,
       headers: {
@@ -53,7 +50,6 @@ function signAndSend(message, name, domain, req, res, targetDomain) {
 
 function sendAcceptMessage(thebody, name, domain, req, res, targetDomain) {
   const guid = crypto.randomBytes(16).toString('hex');
-  console.log(thebody);
   let message = {
     '@context': ['https://www.w3.org/ns/activitystreams', 'https://w3id.org/security/v1'],
     'id': `https://${domain}/${guid}`,
@@ -101,7 +97,6 @@ router.post('/', function (req, res) {
         followers = [req.body.actor];
       }
       let followersText = JSON.stringify(followers);
-      console.log('adding followersText', followersText);
       // update into DB
       db.prepare('update accounts set followers = ? where name = ?').run(followersText, `${name}@${domain}`);
     }

updateFeeds.js

@@ -12,11 +12,10 @@ const beanstalkd = new Jackd();
 
 beanstalkd.connect()
 
-async function foo() {
+async function processQueue() {
   while (true) {
     try {
       const { id, payload } = await beanstalkd.reserve()
-      console.log(payload)
       /* ... process job here ... */
       await beanstalkd.delete(id)
       await doFeed(payload)
@@ -27,7 +26,7 @@ async function foo() {
   }
 }
 
-foo()
+processQueue()
 
 function doFeed(feedUrl) {
 return new Promise((resolve, reject) => {
@@ -63,7 +62,6 @@ return new Promise((resolve, reject) => {
         let brandNewItems = newItems.filter(el => difference.includes(el.guid) || difference.includes(el.title) || difference.includes(el.description));
         let acct = feed.username;
         let domain = DOMAIN;
-        //console.log(acct, brandNewItems);
 
         // send the message to everyone for each item!
         for (var item of brandNewItems) {
@@ -101,7 +99,6 @@ return new Promise((resolve, reject) => {
 // This is a function with a bunch of custom rules for different kinds of content I've found in the wild in things like Reddit rss feeds. Right now we just use the first image we find, if any.
 function transformContent(item) {
   let cheerio = require('cheerio');
-  console.log(JSON.stringify(item));
   if (item.content === undefined) {
     item.urls = [];
     return item;
@@ -111,12 +108,10 @@ function transformContent(item) {
   // look through all the links to find images
   let links = $('a');
   let urls = [];
-  //console.log('links', links.length);
   links.each((i,e) => {
     let url = $(e).attr('href');
     // if there's an image, add it as a media attachment
     if (url && url.match(/(http)?s?:?(\/\/[^"']*\.(?:png|jpg|jpeg|gif|png|svg))/)) {
-      //console.log(url);
       urls.push(url);
     }
   });
@@ -127,7 +122,6 @@ function transformContent(item) {
     let url = $(e).attr('src');
     // if there's an image, add it as a media attachment
     if (url) {
-      //console.log(url);
       urls.push(url);
       // remove the image from the post body since it's in the attachment now
       $(e).remove();
@@ -139,7 +133,6 @@ function transformContent(item) {
   // find iframe embeds and turn them into links
   let iframes = $('iframe');
   iframes.each((i,e) => {
-    console.log('iframe',i,e);
     let url = $(e).attr('src');
     $(e).replaceWith($(`<a href="${url}">[embedded content]</a>`));
   });
@@ -163,10 +156,8 @@ function transformContent(item) {
 // TODO import these form a helper
 function signAndSend(message, name, domain, req, res, targetDomain, inbox) {
   // get the private key
-  console.log('sending to ', name, targetDomain, inbox);
   let inboxFragment = inbox.replace('https://'+targetDomain,'');
   let result = db.prepare('select privkey from accounts where name = ?').get(`${name}@${domain}`);
-  //console.log('got key', result === undefined, `${name}@${domain}`);
   if (result === undefined) {
     console.log(`No record found for ${name}.`);
   }
@@ -184,7 +175,6 @@ function signAndSend(message, name, domain, req, res, targetDomain, inbox) {
     const signature_b64 = signature.toString('base64');
     const algorithm = 'rsa-sha256';
     let header = `keyId="https://${domain}/u/${name}",algorithm="${algorithm}",headers="(request-target) host date digest",signature="${signature_b64}"`;
-    //console.log('signature:',header);
     request({
       url: inbox,
       headers: {
@@ -228,7 +218,6 @@ function createMessage(text, name, domain, item, follower, guidNote) {
 
   // add image attachment
   let attachment;
-  console.log('NUM IMAGES',item.urls.length);
   if (item.enclosure && item.enclosure.url && item.enclosure.url.includes('.mp3')) {
     attachment = {
       'type': 'Document',
@@ -261,7 +250,6 @@ function createMessage(text, name, domain, item, follower, guidNote) {
     out.object.attachment = attachment;
   }
 
-  console.log(guidCreate, guidNote);
   db.prepare('insert or replace into messages(guid, message) values(?, ?)').run( guidCreate, JSON.stringify(out));
   db.prepare('insert or replace into messages(guid, message) values(?, ?)').run( guidNote, JSON.stringify(out.object));
 
@@ -269,11 +257,9 @@ function createMessage(text, name, domain, item, follower, guidNote) {
 }
 
 function sendCreateMessage(text, name, domain, req, res, item) {
-  // console.log(`${name}@${domain}`);
   let result = db.prepare('select followers from accounts where name = ?').get(`${name}@${domain}`);
   let followers = JSON.parse(result.followers);
   const guidNote = crypto.randomBytes(16).toString('hex');
-  // console.log(followers);
   if (!followers) {
     followers = [];
   }

views/home.pug

@@ -8,4 +8,5 @@ html
     h1 RSS to ActivityPub Converter
     p.account by <a href="https://friend.camp/@darius">Darius Kazemi</a>
     p This is a service to convert any RSS feed into an account that Mastodon (or any other ActivityPub social network) can subscribe to. <a href="https://github.com/dariusk/rss-to-activitypub/">Source code here</a>.
-    p <a href="/convert/">Click here to start!</a>
+    p Only users of <a href="https://#{OAUTH.domain}">#{OAUTH.domain_human}</a> can create feeds, but anyone can subscribe if they know the account info.
+    p <a href="https://#{OAUTH.domain}#{OAUTH.authorize_path}?scope=read:accounts&response_type=code&redirect_uri=#{OAUTH.redirect_uri}&client_id=#{OAUTH.client_id}">Click here to log in!</a>