Obsidian + đŸȘŽ Quartz 4.0

DST Lezione 18

8 min read

DST Lezione 18

Sources:

Cost of data management

The source of this lecture is the specifici data management costs from the university of uthrect but it gives a general idea on what it is cost of data management and its components.

Data Brokers

Data has a price, and the most valuable data is from user profiling. Data brokers are the ones that buy and sell this data. For any kind of business activity where you have clients and you want to adapt your business to your clients, you need data profiling.

An example scenario

Say we have a big company that sell a service through a platform, for example that sell films. You have three main component:

  • Content/service offered from the company
  • A platform where clients can access contents
  • The Clients that interacts with the platform that mediates access to content.

This is a typical scenario. The first two question to ask are:

  • Is your service data intensive or not?
  • Is your service computational intensive or not? How much data and how much computational power you need are core variables.

A typical business would earn money not mainly from the clients using the platform (i.e premium content, subscription), but from advertising. Your clients aren’t (only) the users, but (also) the advertisers. To sell more advertisers and get more money you need the clients to stay more on the platform, incresing the engagement time. To increase your sales, you need a recomendation systems such that each client get personalized content & advs. The better is your recomendation system, the more effective recomendations and the more money you get from advertisers. That’s one of the most exploited application of AI algorithms.

Netflix challenge and confermation bias

Netflix challenge

Years ago, Netflix offered a $1 billion challenge in scientific computing. They provided a large matrix with users as rows and films as columns, containing user ratings from 0 to 5 or missing. They then gave some films without ratings and asked for the best algorithm to predict these ratings. The true values were known to Netflix, but only the training set was given to participants.

This challenge was won by some scientists. The lesson here is that with enough training data and a good machine learning algorithm, you can predict users’ preferences for content accurately.

Netflix took this further by creating movies based on data they have, they already know what people like so this reduces the risk to almost zero. This is why movie workers are striking; they’re being replaced by AI. Economically, it makes sense for Netflix.

Confermation Bias

Recommendation systems show people what they want to see, creating a bubble where users only see content that aligns with their existing views. This isn’t just a problem in politics; it happens in all areas. It’s a consequence of widely using recommendation systems that adapt to individual preferences.

An example scenario - part 2

You have three sources of data in this example scenario that can be used to build a recomendation system:

  • Anagraphic / demographic data from users that is required because they subscribed to the platform. They can be clustered to get homogenous users group.
  • Content data like film, they can be clustered and classified.
  • Interaction from users in the platform: you know which user has given what rating to what movie.

Many people are equal to each other and with so many data you can predict even for users that “you don’t know”. Years ago content classification was done by hand.

With access to multiple platforms like Meta, you can cross-reference data to get even more accurate predictions. For example, social network interactions can be used to group users by how much they interact, clustering them by affinity. This is known as behavioral data.

On Social Media Survey

Studying sample theory reveals that social media surveys often lack statistical validity because they likely include a homogenous group of people. However, the vast amount of data on social media can still be used to study and influence political opinions.

Obtaining existing data with scraping

Usually, you’d hit up a data broker for info. If you want to do it alone, you could do scraping. One source of scraping could be the web.

But there is a problem. Let’s say you want to create an automatic classifier for resumes to predict if someone’s a reliable employee. You’d use resume data as your source, right? But if you just grab resumes from Google, any biases in that data get passed on to your classifier. Any biases that lies in the data is transfered to the classifier and could even make biases worse. For instance, if most resumes are from male computer scientists, your classifier might unfairly favor men. For good data you have to pay.

For example in researching, sometimes you have to train people to conduct interviews for quality data. About the curriculum example, there is a whole research field if you’re interested.

Data Storage

If you want to estimate data storage you cannot just say the volume on my data is 1TB, so i need 1TB storage. You need to know how to classify hot data, warm data, frozen data. What is the portion of data that will be accessed the most frequently? What is the data that is accessed more stored?

Assume 1GB for film for simplicity and you have 100TB of movies. That just the size but you will not use the same memory for all of them. There will be a small portions that are hot and most of them will be seen very rarely.

Remember the hierarchy of data: a pyramid with hot, warm,cold, frozen data - in terms of request. The vertex of the pyramid is the small fraction in proportion to the rest. Then you have the hierarchy of memory, and this is low latency high price per byte. You want this to be map to correspond, so you want frozen data to be stored in high latency. You cannot use just one system.

When estimating data storage, it’s not as simple as saying, “My data is 1TB, so I need 1TB of storage.” You’ve gotta figure out how to categorize your data—what’s hot, what’s warm, what’s frozen. What fraction of your data gets accessed the most frequently? What’s just sitting around, hardly ever touched? (Data Temperature)

Let’s say you’ve got 100TB of movies, each about 1GB. But you’re not gonna use the same storage for all of ‘em. There’s gonna be a small chunk that’s hot and gets seen a lot, and then there’s the rest, hardly ever touched.

Data anonimization

Sometime you cannot use data that is in clear but you have to encrypt. The tech solution is anonimization of data.

For example say , is an interaction between an user and a movie. The most valuable information is the interaction, i don’t care the user name, so i could mask the user data and i could apply the algorithm the same way and comply to the law.

“Even if you don’t use the system, there will be tons of people like you and they could still predict based on them being on the platform. “

Data preservation and sharing

(Sta riassumendo da quel link che ha mandato) https://www.uu.nl/en/research/research-data-management/guides/costs-of-data-management

When you need to prepare data for preservation and sharing you need to consider the following extra costs:

  • Data cleaning, if not done already
  • Digitizing analogue or paper data. Here it depends if it can be done with an Optical Character Recognition or if you need to scan special materials thus using different approaches
  • Converting data to sustainable file formats
  • The cost of uploading a dataset into a data repository. For example Dryad costs 150 dollars per dataset.

You can search for a suitable data repository in our Data repository finder or in the Registry of Research data Repositories.

Hiring Expertise

Another costs comes from hiring expertise. You most likely cannot do everything alone, and there may be some stuff that require external experts on the matter.

Graph View

Backlinks