Difference between revisions of "Cross-site authenticated RSS"

From Dreamwidth Notes
Jump to: navigation, search
(New page: Category: Brainstorming Or: How we can make it so that people can read friends-only entries on other sites from their DW reading page. I dump in an email I wrote in order to kick off...)
 
m
 
(9 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
[[Category: Brainstorming]]
 
[[Category: Brainstorming]]
 +
[[Category:Feeds]]
  
 
Or: How we can make it so that people can read friends-only entries on other sites from their DW reading page.
 
Or: How we can make it so that people can read friends-only entries on other sites from their DW reading page.
Line 5: Line 6:
 
I dump in an email I wrote in order to kick off the conversation:
 
I dump in an email I wrote in order to kick off the conversation:
  
−
<pre>Syndicated feeds are treated like any other account -- each item in the feed is given its own entryID in the syndicated account, and that behaves just like any other entry (in terms of building the friends page, etc). Now, there's a mechanism in RSS to authenticate to the server providing the feed, and LJ treats that authenticated RSS like they would if you were logged in; if you'd see the entry on someone else's journal, you'll see it in their RSS feed while you're using RSS authentication.
+
<pre>Syndicated feeds are treated like any other account -- each item in the feed
 +
is given its own entryID in the syndicated account, and that behaves just like
 +
any other entry (in terms of building the friends page, etc). Now, there's a
 +
mechanism in RSS to authenticate to the server providing the feed, and LJ
 +
treats that authenticated RSS like they would if you were logged in; if you'd
 +
see the entry on someone else's journal, you'll see it in their RSS feed while
 +
you're using RSS authentication.
  
−
The easy-but-evil solution would be for us to use someone's authentication credits to pull down the RSS feed of a journal, store all the FO entries locally, and then every time someone who had that feed friended reloaded their FP, check against the feedowner's friends list on LJ to see if the local viewer was permitted to see that entry. That's the evil way to do it. We aren't going to do that.
+
The easy-but-evil solution would be for us to use someone's authentication
 +
credits to pull down the RSS feed of a journal, store all the FO entries
 +
locally, and then every time someone who had that feed friended reloaded their
 +
FP, check against the feedowner's friends list on LJ to see if the local
 +
viewer was permitted to see that entry. That's the evil way to do it. We
 +
aren't going to do that.
  
−
Now, the idea that I'm currently fronting to solve the problem is instead to create a new type of account -- call it an aggregation account or whatever. Users can then give us the list of people who have them friended on LJ (or wherever, assume that I mean any LJ-based site when I say LJ, it's easier) and also give us their login credentials for LJ, or give us their LJ OpenID or whatever, any way that we can get the authentication done, I'm not a coder, blah blah. The aggregation account would then poll those RSS feeds at a MUCH slower rate than for the regular RSS feed (to prevent LJ from banning us for hitting them too often -- because with a regular syndication account for just the public entries, you only have to check once; to check for FO entries, you have to check with every single person's separate credentials to see if they have entries they can see) and see if there are any FO entries that they can see.
+
Now, the idea that I'm currently fronting to solve the problem is instead to
 +
create a new type of account -- call it an aggregation account or whatever.
 +
Users can then give us the list of people who have them friended on LJ (or
 +
wherever, assume that I mean any LJ-based site when I say LJ, it's easier) and
 +
also give us their login credentials for LJ, or give us their LJ OpenID or
 +
whatever, any way that we can get the authentication done, I'm not a coder,
 +
blah blah. The aggregation account would then poll those RSS feeds at a MUCH
 +
slower rate than for the regular RSS feed (to prevent LJ from banning us for
 +
hitting them too often -- because with a regular syndication account for just
 +
the public entries, you only have to check once; to check for FO entries, you
 +
have to check with every single person's separate credentials to see if they
 +
have entries they can see) and see if there are any FO entries that they can
 +
see.
  
−
If there is a new entry, and it's FO/filtered, and the person whose login credentials are being supplied can see it, we insert an item into the aggregation account that says "your friend $friendname posted a friends-locked entry that you can see. Its URL is: $url".
+
If there is a new entry, and it's FO/filtered, and the person whose login
 +
credentials are being supplied can see it, we insert an item into the
 +
aggregation account that says "your friend $friendname posted a friends-locked
 +
entry that you can see. Its URL is: $url".
  
−
The aggregation account will then insert that item into the actual reading list, so that notification will appear on your DW readlist -- not the *content* of the entry, but the fact that someone *posted* it, along with a link. Sure, there's a chance the poster will have re-filtered the entry by the time the person gets to it, especially since we'd *have* to make the polling-for-FO-entries a very slow thing (probably only once every 12 or 24 hours), but we could still introduce the fact that there *was* an entry into your reading list. You could then go click the link and open up the entry in another tab, another window, etc.</pre>
+
The aggregation account will then insert that item into the actual reading
 +
list, so that notification will appear on your DW readlist -- not the
 +
*content* of the entry, but the fact that someone *posted* it, along with a
 +
link. Sure, there's a chance the poster will have re-filtered the entry by the
 +
time the person gets to it, especially since we'd *have* to make the
 +
polling-for-FO-entries a very slow thing (probably only once every 12 or 24
 +
hours), but we could still introduce the fact that there *was* an entry into
 +
your reading list. You could then go click the link and open up the entry in
 +
another tab, another window, etc.</pre>
 
--[[User:Rahaeli|Rahaeli]] 10:27, 30 January 2009 (UTC)
 
--[[User:Rahaeli|Rahaeli]] 10:27, 30 January 2009 (UTC)
 +
 +
==Alternate Presentation==
 +
This presentation seems clunky to me (or rather, I personally detest URL only RSS feeds.) I wonder if we could gracefully reference the desired text (possibly in ?format=light) in an iframe. I know iframes aren't usable with screen readers and the like, but in the case where the browser doesn't support frames, doesn't the standard have a NOFRAME where we could put just the "your friend $friendname posted a friends-locked entry that you can see. Its URL is: $url" line. This would have to be aware of all lj-clone sites so we would know to append ?format=light to some URLs and not others.
 +
--[[User:Zvi-likes-tv|Zvi-likes-tv]] 14:32, 2 February 2009 (UTC)
 +
 +
A different thought that I had at Synecdochic's last Stitch & Bitch. I don't know if it's being shared somewhere not the wiki or not, so I will record it for posterity:
 +
 +
* Given that the closed entry polling is going to have to occur at long intervals, which will, I assume, play merry havoc with the ordering of the flist
 +
* Given that the plan is ''not'' to store the locked content on DW's servers, but merely give a URL or, perhaps, incorporate through frames, and this will play merry havoc with people's styles
 +
 +
Perhaps the best plan is not to integrate locked posts with the reading page, but instead push it to the inbox as tracking. I don't know (and suspect it depends greatly on backend stuff I know nothing of) if it would be better to keep the update to once a day or hit a user's inbox every time DW discovers a locked entry the user has access to.
 +
 +
The two most obvious cons I can think of to this plan are that (a) it's a completely new behavior and (b) it uses up a precious tracking subscription for free users, who only have 25 to start.
 +
 +
--[[User:Zvi-likes-tv|Zvi-likes-tv]] 00:32, 20 February 2009 (UTC)
 +
 +
==Persistent Login==
 +
My base assumption is that only Bad Guys want me to give them my password to another account. I am just sayin'. --[[User:Zvi-likes-tv|Zvi-likes-tv]] 14:32, 2 February 2009 (UTC)
 +
 +
==Make it DW-only and customizable==
 +
 +
It seems to me that we'd be better off designing this so that different DW-based sites can read friends-only locked posts from one another on the friends page, and then provide an API that other sites (LJ) could use if they wanted to support it.  That way we can make it work right, instead of trying to hack around an implemenation that isn't designed to do what we want it to do.
 +
 +
For instance, you could get a reasonably elegant solution if you just extended OpenID and RSS a little bit.  If you extend OpenID so that you could send a request authenticated as multiple users (I assume you can't do that now), then a DW server could connect to a feed and authenticate as all of the local users who watch that feed.  Then, you could use an extended RSS namespace (which is already supported in RSS 2.0) to add in a list of the authenticated users who could access each of the entries, or show them as pubic if anyone can read them.  Then you store the privacy settings for each entry in the feed, and only show locked entries to the appropriate users.
 +
 +
There's already a preference setting saying how to format your RSS feed (title only, summary, full post), so it wouldn't be that bad to extend it to have a separate setting for locked posts. That way if a user is ok with sharing their full posts with the remote servers, they can, and if not, they can just send a link.
 +
 +
The RSS namespace extension would be easy.  Storing the allow list for locked feed entries shouldn't be _too_ bad.  Extending OpenID might be a bit trickier...  --[[User:Allen|Allen]] 22:18, 3 February 2009 (UTC)
 +
 +
: How does the originating site notify the polling site of edits to the entry? How does the originating site notify the polling site of edits to the trust group? I think there's a problem that (a) the friendslist promises to be very current and (b) a locked entry implies the writer of it wants very fine-grained control, which we should respect.
 +
 +
:Also, I'm not sure that implementing this as if the problem is one of clones talking to each other is the right call. I suspect a large number of people will use their reading list as an RSS reader, and I think it's a good idea to support the authenticated protocol more generally across the Internet. Especially since the barrier to initial adoption is going to be getting people to come '''from''' LJ clones to DW. If it's DW only, it's not useful to people moving here from LJ/IJ/DJ, who haven't managed to convince their friendslist to move with them. --[[User:Zvi-likes-tv|Zvi-likes-tv]] 08:18, 6 March 2009 (UTC)
 +
 +
==Why we can't just have individual syndicated feeds for each DW user ==
 +
 +
1). OtherService would ban us in a heartbeat for hitting them too much. (I have something like two thousand people reading me. Instead of one feed polling every 30 minutes, that'd be 2000 feeds polling every 30 minutes. Now multiply that by a whole lot of users.) The lower time interval between polling for cross-site authenticated entries is designed to slow that down.
 +
 +
2). We would then be storing other people's protected content on Dreamwidth. That makes a lot of people go "euuuuuugh", including us.
 +
 +
3). People edit entry security after posting. We'd have to *re*-poll OtherService every time someone looked at their reading page if it had an entry from a syndicated feed to make sure that they still have credentials to see it.
 +
 +
4). People take other people off their friends list. We'd have to re-poll OtherService every time someone looked at their reading page if it had an entry from a syndicated feed to make sure they're still on that list.
 +
 +
5). We'd be storing thousands of duplicate entires on our side, and assuming that most people post a mix of public and FO entries on OtherSite, that'd be a massive duplication of effort on our disks. --[[User:Rahaeli|Rahaeli]] 07:13, 4 February 2009 (UTC)
 +
 +
== Sad Project News (Update from <dwcomm>dw_news</dwcomm> on 2010 Dec 24) ==
 +
One of the things we've been trying to accomplish since the open beta launch is the ability to read your offsite friends page/reading lists from other LJ-based sites directly on Dreamwidth itself, along with things like locked posts (assuming you have the credentials to read them, of course). It's been a while since we've given you an update on that one, and there's a reason for that -- the project has had a bunch of setbacks. [staff profile] mark actually had working code, and could have committed it at any point in the past, oh, nine months ... except for some factors that meant the near-certainty that the minute we turned it on, LiveJournal will block our access to their servers.
 +
 +
We have been looking for a way to work around that problem -- rate-limiting our queries, further limiting who got access to the feature, even throwing out all the code Mark had written and starting again from scratch, but nothing we did could get around the fact that if we went ahead with the implementation, we'd be almost guaranteed to be blocked from accessing LJ entirely. That would mean not only no more cross-site reading list, but also no importing, no crossposting, and no syndicated feeds from LiveJournal. The past nine months or so have been full of a lot of behind-the-scenes brainstorming on how we could make things work, since we know how much people have been looking forward to the prospect, but nothing has worked and we've hit the bottom of the barrel on other things we can try.
 +
 +
It is with utmost reluctance that I have to announce that unless something changes drastically in the future, we aren't going to be able to offer the cross-site reading list feature after all. We'd rather keep importing, crossposting, and syndicated feeds (and not offer cross-site reading) than not be able to offer any interaction with LJ at all. We are all incredibly sorry about this -- we genuinely thought we'd be able to do it, but some factors changed after we first announced the project and no matter what we try, we can't work around them.
 +
 +
For those who are looking for something that will let you read offsite content that requires you to identify yourself in order to see locked data, our best suggestion is to download a desktop RSS feedreader that supports HTTP authentication. (I don't know of any online feedreader that will do it, but many desktop ones will.)
 +
 +
We are absolutely going to keep this problem in mind, and if the factors that are keeping us from being able to release this feature ever change, or if LJ implements a better way for their users to access their data through the LJ protocol, we will go back to it in a heartbeat.
 +
 +
--<dwuser>denise</dwuser> @ 2010-12-24 07:05 pm in http://dw-news.dreamwidth.org/27853.html

Latest revision as of 17:57, 17 March 2013


Or: How we can make it so that people can read friends-only entries on other sites from their DW reading page.

I dump in an email I wrote in order to kick off the conversation: 

Syndicated feeds are treated like any other account -- each item in the feed
is given its own entryID in the syndicated account, and that behaves just like
any other entry (in terms of building the friends page, etc). Now, there's a
mechanism in RSS to authenticate to the server providing the feed, and LJ
treats that authenticated RSS like they would if you were logged in; if you'd
see the entry on someone else's journal, you'll see it in their RSS feed while
you're using RSS authentication.

The easy-but-evil solution would be for us to use someone's authentication
credits to pull down the RSS feed of a journal, store all the FO entries
locally, and then every time someone who had that feed friended reloaded their
FP, check against the feedowner's friends list on LJ to see if the local
viewer was permitted to see that entry. That's the evil way to do it. We
aren't going to do that.

Now, the idea that I'm currently fronting to solve the problem is instead to
create a new type of account -- call it an aggregation account or whatever.
Users can then give us the list of people who have them friended on LJ (or
wherever, assume that I mean any LJ-based site when I say LJ, it's easier) and
also give us their login credentials for LJ, or give us their LJ OpenID or
whatever, any way that we can get the authentication done, I'm not a coder,
blah blah. The aggregation account would then poll those RSS feeds at a MUCH
slower rate than for the regular RSS feed (to prevent LJ from banning us for
hitting them too often -- because with a regular syndication account for just
the public entries, you only have to check once; to check for FO entries, you
have to check with every single person's separate credentials to see if they
have entries they can see) and see if there are any FO entries that they can
see.

If there is a new entry, and it's FO/filtered, and the person whose login
credentials are being supplied can see it, we insert an item into the
aggregation account that says "your friend $friendname posted a friends-locked
entry that you can see. Its URL is: $url".

The aggregation account will then insert that item into the actual reading
list, so that notification will appear on your DW readlist -- not the
*content* of the entry, but the fact that someone *posted* it, along with a
link. Sure, there's a chance the poster will have re-filtered the entry by the
time the person gets to it, especially since we'd *have* to make the
polling-for-FO-entries a very slow thing (probably only once every 12 or 24
hours), but we could still introduce the fact that there *was* an entry into
your reading list. You could then go click the link and open up the entry in
another tab, another window, etc.

--Rahaeli 10:27, 30 January 2009 (UTC)

Alternate Presentation

This presentation seems clunky to me (or rather, I personally detest URL only RSS feeds.) I wonder if we could gracefully reference the desired text (possibly in ?format=light) in an iframe. I know iframes aren't usable with screen readers and the like, but in the case where the browser doesn't support frames, doesn't the standard have a NOFRAME where we could put just the "your friend $friendname posted a friends-locked entry that you can see. Its URL is: $url" line. This would have to be aware of all lj-clone sites so we would know to append ?format=light to some URLs and not others. --Zvi-likes-tv 14:32, 2 February 2009 (UTC)

A different thought that I had at Synecdochic's last Stitch & Bitch. I don't know if it's being shared somewhere not the wiki or not, so I will record it for posterity:

  • Given that the closed entry polling is going to have to occur at long intervals, which will, I assume, play merry havoc with the ordering of the flist
  • Given that the plan is not to store the locked content on DW's servers, but merely give a URL or, perhaps, incorporate through frames, and this will play merry havoc with people's styles

Perhaps the best plan is not to integrate locked posts with the reading page, but instead push it to the inbox as tracking. I don't know (and suspect it depends greatly on backend stuff I know nothing of) if it would be better to keep the update to once a day or hit a user's inbox every time DW discovers a locked entry the user has access to.

The two most obvious cons I can think of to this plan are that (a) it's a completely new behavior and (b) it uses up a precious tracking subscription for free users, who only have 25 to start.

--Zvi-likes-tv 00:32, 20 February 2009 (UTC)

Persistent Login

My base assumption is that only Bad Guys want me to give them my password to another account. I am just sayin'. --Zvi-likes-tv 14:32, 2 February 2009 (UTC)

Make it DW-only and customizable

It seems to me that we'd be better off designing this so that different DW-based sites can read friends-only locked posts from one another on the friends page, and then provide an API that other sites (LJ) could use if they wanted to support it. That way we can make it work right, instead of trying to hack around an implemenation that isn't designed to do what we want it to do.

For instance, you could get a reasonably elegant solution if you just extended OpenID and RSS a little bit. If you extend OpenID so that you could send a request authenticated as multiple users (I assume you can't do that now), then a DW server could connect to a feed and authenticate as all of the local users who watch that feed. Then, you could use an extended RSS namespace (which is already supported in RSS 2.0) to add in a list of the authenticated users who could access each of the entries, or show them as pubic if anyone can read them. Then you store the privacy settings for each entry in the feed, and only show locked entries to the appropriate users.

There's already a preference setting saying how to format your RSS feed (title only, summary, full post), so it wouldn't be that bad to extend it to have a separate setting for locked posts. That way if a user is ok with sharing their full posts with the remote servers, they can, and if not, they can just send a link.

The RSS namespace extension would be easy. Storing the allow list for locked feed entries shouldn't be _too_ bad. Extending OpenID might be a bit trickier... --Allen 22:18, 3 February 2009 (UTC)

How does the originating site notify the polling site of edits to the entry? How does the originating site notify the polling site of edits to the trust group? I think there's a problem that (a) the friendslist promises to be very current and (b) a locked entry implies the writer of it wants very fine-grained control, which we should respect.
Also, I'm not sure that implementing this as if the problem is one of clones talking to each other is the right call. I suspect a large number of people will use their reading list as an RSS reader, and I think it's a good idea to support the authenticated protocol more generally across the Internet. Especially since the barrier to initial adoption is going to be getting people to come from LJ clones to DW. If it's DW only, it's not useful to people moving here from LJ/IJ/DJ, who haven't managed to convince their friendslist to move with them. --Zvi-likes-tv 08:18, 6 March 2009 (UTC)

Why we can't just have individual syndicated feeds for each DW user

1). OtherService would ban us in a heartbeat for hitting them too much. (I have something like two thousand people reading me. Instead of one feed polling every 30 minutes, that'd be 2000 feeds polling every 30 minutes. Now multiply that by a whole lot of users.) The lower time interval between polling for cross-site authenticated entries is designed to slow that down.

2). We would then be storing other people's protected content on Dreamwidth. That makes a lot of people go "euuuuuugh", including us.

3). People edit entry security after posting. We'd have to *re*-poll OtherService every time someone looked at their reading page if it had an entry from a syndicated feed to make sure that they still have credentials to see it.

4). People take other people off their friends list. We'd have to re-poll OtherService every time someone looked at their reading page if it had an entry from a syndicated feed to make sure they're still on that list.

5). We'd be storing thousands of duplicate entires on our side, and assuming that most people post a mix of public and FO entries on OtherSite, that'd be a massive duplication of effort on our disks. --Rahaeli 07:13, 4 February 2009 (UTC)

Sad Project News (Update from [info]dw_news on 2010 Dec 24)

One of the things we've been trying to accomplish since the open beta launch is the ability to read your offsite friends page/reading lists from other LJ-based sites directly on Dreamwidth itself, along with things like locked posts (assuming you have the credentials to read them, of course). It's been a while since we've given you an update on that one, and there's a reason for that -- the project has had a bunch of setbacks. [staff profile] mark actually had working code, and could have committed it at any point in the past, oh, nine months ... except for some factors that meant the near-certainty that the minute we turned it on, LiveJournal will block our access to their servers.

We have been looking for a way to work around that problem -- rate-limiting our queries, further limiting who got access to the feature, even throwing out all the code Mark had written and starting again from scratch, but nothing we did could get around the fact that if we went ahead with the implementation, we'd be almost guaranteed to be blocked from accessing LJ entirely. That would mean not only no more cross-site reading list, but also no importing, no crossposting, and no syndicated feeds from LiveJournal. The past nine months or so have been full of a lot of behind-the-scenes brainstorming on how we could make things work, since we know how much people have been looking forward to the prospect, but nothing has worked and we've hit the bottom of the barrel on other things we can try.

It is with utmost reluctance that I have to announce that unless something changes drastically in the future, we aren't going to be able to offer the cross-site reading list feature after all. We'd rather keep importing, crossposting, and syndicated feeds (and not offer cross-site reading) than not be able to offer any interaction with LJ at all. We are all incredibly sorry about this -- we genuinely thought we'd be able to do it, but some factors changed after we first announced the project and no matter what we try, we can't work around them.

For those who are looking for something that will let you read offsite content that requires you to identify yourself in order to see locked data, our best suggestion is to download a desktop RSS feedreader that supports HTTP authentication. (I don't know of any online feedreader that will do it, but many desktop ones will.)

We are absolutely going to keep this problem in mind, and if the factors that are keeping us from being able to release this feature ever change, or if LJ implements a better way for their users to access their data through the LJ protocol, we will go back to it in a heartbeat.

--[info]denise @ 2010-12-24 07:05 pm in http://dw-news.dreamwidth.org/27853.html

Retrieved from "http://wiki.dreamwidth.net/wiki/index.php?title=Cross-site_authenticated_RSS&oldid=11207"