2 posts tagged “data collection”

We can see that there are three main models: user, photoset, and photo. The models are interlinked with the following possibilities:

• photoset can link to { photo, user }
• photo can link to { photoset, user }
• user can link to { photoset, photo, user }
  

A more detailed description of these linkages are:

• photoset links to
  
• users (as commenters)
• photos (as children)
• user (as owner)
• photo links to
  
• photosets (as parents)
• users (as favoriters)
• users (as commenters)
• users (as noters)
  
• users (as taggers)
• user links to
  
• photosets (as children)
  
• photos (as children)
  
• users (as contacts)
  
• photos (as favorites)
  

With the three main "concepts" linking to each other, i.e. users, photosets, and photos, we face a problem of how deep to traverse the linkages. One approach is to impose some kind of degree or hop count. Here, we can define a limit of degree k to mean that the traversal will stop going deeper if it reaches any model for the kth time.

Thus, for k = 2 and since our starting point is always a photoset, we have:

• photoset -> user -> photoset, i.e. photosets owned by the user(s) who (commented|owned) the photoset
• photoset -> user -> user, i.e. users who are contacts of the user who (commented|owned) the photoset
• photoset -> user -> photo -> user, i.e. users who (favorited|commented|noted|tagged) photos (belonging to|favorited by) user(s) who (commented|owned) the photoset
• photoset -> user -> photo -> photoset,  i.e. photosets that contain the photos (belonging to|favorited by) user(s) who (commented|owned) the photoset
• photoset -> photo -> photoset, i.e. other photosets that contain the same photos as the photoset
• photoset -> photo -> user -> user, i.e. users who are contacts of the user who (favorited|commented|noted|tagged) photos belonging to the photoset
• photoset -> photo -> user -> photo, i.e. photos (belonging to|favorited by) users who (favorited|commented|noted|tagged) photos belonging to the photoset
• photoset -> photo -> user -> photoset, i.e. photosets belonging to users who (favorited|commented|noted|tagged) photos belonging to the photoset

So Ive basically done the following for the past few days (since Wednesday):

1. Enumerate all functions in flickr as a table. The webpages they provided was not useful for inter-function comparisons
2. Trimmed down the api to just the functions I need / can use (most require user authentication and was not usable)
3. And analyzed the responses from those functions
   
4. Designed a schema for storing the data
5. Designed the program flow that fetches the data

Next is to actually write the program. I've decided to script it with ruby and have the data stored as flat files, i.e. there will be one file for each table. The files will probably be CSV or TSV with the first line listing the attribute names. Since field values can also be data structures and not mere strings, e.g. Hashes, Arrays, etc, I will encode all field values into JSON. As for the photo files, they will be stored as a set of photoset folders. The filenames will be the photo_id. The folder name will be the photoset_id.