Are databases significantly more efficient than text files for storing data?

For large amounts of data, say chat logs.

It depends on what exactly is the data, and how you are going to use it.

If they’re chatlogs, and you don’t plan on retrieving it from anywhere else besides in-game via some administration panel; I’d store them in a text file.

Proper databases are easier to maintain and have the advantages of strong data integrity. Use them.

Use txt files and intelligently name the files. You don’t need a database for that.

Well they could be chat logs, that was really just an example. It could also be something like SteamID’s, Vectors, or anything else worth storing. In terms of overall read/write times and processor efficiency, which is better? or are they comparable? This is especially considering once you get into storing lots and lots of data.

Generally, databases are the best solution for storing stuff that you will have to modify later on, especially when you have a bunch of different changes going (like when you’re doing changes for multiple players over time). Even huge logs in databases are pretty efficient when you use the ARCHIVE table format in MySQL.

txt is faster unless you have a giant table.

fileio is extremely expensive. Databases will be faster in terms of that and also depending on the data and how the data is acquired.

Depending on the data, how you retrieve it ( Is it networked data or is it known information? ) makes a difference as well. If you have to network the data, it’ll take additional time, regardless of whether or not you’re using a database.

I agree with previous posts; chat-logs should be stored on the server with intelligent names. If the client wants a copy, you could easily allow the client to save their logs, or make it automated.

If you’re storing account information, characters, etc; use a database. It’ll be faster and you can do more with it in terms of storing less data, id referencing, table / data-set linking.

If you store data in a text file, you will have to store it in a way that can be processed. A common way in Garry’s Mod / Source is to use JSON. You can convert tables to string, and back again. This means you have to process it to read, and to save, each and every time. If the file write fails and you don’t have a way to solve it, the data is corrupted and lost.

There’s more, but I have a massive headache and am not feeling well.

Alrighty thanks a bunch. It seems like doing everything through the database is just the better option, especially since the data is a lot more accessible through SQL queries.

Databases store their data in files as well though.

Also, there’s the additional overhead/code bloat from maintaining an open socket connection and having to escape the strings. Unless you need to search the chatlogs or access them from somewhere else than the server using flatfiles is probably a better idea.

You’re right, it does store in files. But the manner in which it does it, coupled with the additional way to retrieve data is much quicker. You don’t have to parse the data yourself, etc. Google even uses a modified version of MySQL for their search-engine which shows you how fast it can be, to sift through billions, if not trillions, of records.

Oh no no no, I mean storing files in the built in SQLite database, not an external server.

You’re right, you don’t.

There’s a ridiculous number of arguments for either case really, most of it just comes down to what you actually need the data for (for example to determine frequency of read/write cycles, whether you need to search it or not and so on).

As I haven’t actually tested that (and I’m sure it varies on a lot more factors than just whether or not you have a “giant table”), so I can’t say anything about it, however; personally, I usually favor elegant and powerful methods over performance (if I ever need to optimize something, the first thing I look at is the algorithm and not whether I can do some ugly tricks such as this), which you might even see reflect in some of my posts.

In this case though, him having to create logs makes me think that I doubt he’s going to access them particularly often and thus the question of speed becomes primarily silly, so I believe that he should choose the method that he is most comfortable working with instead of focusing on what is the most “efficient”.