[QUOTE=phygon;52339182]Man, fuck every single person that leaves a one-star review on a product before reaching out to customer support just to put more pressure on the dev to fix it that second (even more so when the issue is not RTFM). It's seriously not cool, especially when you [I]never fucking respond to the support that you received or take down the review[/I].
In other news, making unity assets is profitable and stressful.[/QUOTE]
That's why I hate rating systems like that. They're utterly pointless because people use it wrong. They either give 5 stars if they like the product even just a bit, or immediately give 1 star is something is a bit off, but never inbetween.
I usually post Half-Payne stuff in HL thread but I think this stuff is also worth it here.
I've got Imgui embedded in Half-Life, here's final touches to new game mode menu
[img]https://pbs.twimg.com/media/DCCRf5fXgAE7Tj3.jpg[/img]
[QUOTE=geel9;52333352]MongoDB was a mistake.[/QUOTE]
SQL DBs are such nice databases, I feel relaxed when I write SQL.
[QUOTE=phygon;52339182]In other news, making unity assets is profitable and stressful.[/QUOTE]
When in a gold rush, sell shovels and pans
[QUOTE=geel9;52335667]Specifically, we're going with MariaDB, not MySQL. We chose that because it's what we're familiar with on scrap.tf and Postgres had enough differences that I felt it wasn't worth learning an entirely new database -- especially when going with what we thought was a "cool choice" for a database back in 2013 turned out to be a terrible move. Frankly, the important part is getting back onto SQL -- the exact engine isn't incredibly relevant.
We aren't doing a seamless transition because that's a lot more work for relatively little payoff. The time to migrate the database is like 30 minutes. That's a very easy sacrifice to make when compared to the hours of time it would take to manage multiple databases simultaneously.[/QUOTE]
Can you tell us something about what made MongoDB a bad choice for you guys versus what SQL does for you?
[QUOTE=Ac!dL3ak;52335737]What happens if something breaks while you're performing the migration?[/QUOTE]
Are you referring to data loss or the website code breaking?
[editline]11th June 2017[/editline]
[QUOTE=ryankingstone;52340779]Can you tell us something about what made MongoDB a bad choice for you guys versus what SQL does for you?[/QUOTE]
Sure.
Marketplace.tf is an e-commerce website for TF2 items. It was launched about a year after we launched its sister (and much larger) website, scrap.tf. Scrap.tf runs on MySQL and always has.
We chose MongoDB initially because we were interested in learning a new technology and weren't expecting Marketplace.tf to become the largest tf2 ecommerce website (besides the SCM). Its NoSQL design interested us; not having to worry about a schema seemed very handy.
So, there are a few main issues with MongoDB. The first is kind of our fault -- we've been using it like a relational database. We have collections that have fields that point to IDs in other collections. MongoDB isn't really supposed to be relational -- that's the whole point. However, the alternative is to store data in multiple places; for instance, if you have a collection called "threads" (forum threads) and a collection called "users", you might store all the thread's posts in a "posts" array in the threads collection, but you might also store all a user's posts in a "posts" array in the users collection. So now you don't have to do any relational searches to find a user's posts, and if you store user data in the threads collection, you don't need to query to find user name/avatar/etc. That's kind of cool, except now whenever you edit a post, you have to edit it in multiple places. And if you edit a user's data -- like their name, name color, avatar, etc. -- you have to edit [i]that[/i] in multiple places. This seems to be the preferred MongoDB setup, and we don't like that, so we don't do it.
This improper relational usage causes us to make more queries than most MongoDB-backed websites likely would for a single page. This makes Mongo's performance worse. Mongo, in general, has pretty bad performance. It's not linearly bad -- you can churn through a million documents and aggregate them pretty quickly, relatively speaking. But it has a lot of overhead. So the fastest queries are going to take longer than the fastest queries on MySQL. At least in our experience.
Also, MongoDB has no joins. Well, okay, it has joins. They're just one of the largest pains to use, and they're not incredibly powerful. In order to join two collections together, you need to use the aggregation pipeline (more on that later). This is a bitch, and AFAIK you can't do multiple joins in the same query (I could be wrong, though.)
MongoDB has terrible defaults. I mean, atrociously bad. By default, it runs without authentication enabled. This is... somewhat acceptable. What's not acceptable is that [i]if there are users defined[/i], authentication still won't be enabled. So you might create a user, set its password and roles properly, and then tell your applications to use the users you created. It'll all work great, except if authentication still isn't enabled, anyone can login using any user/password (or none) and have complete access to your DB. It would make a lot more sense that MongoDB would deny a connection if authentication were disabled but user/password was provided. That makes a lot of sense. This isn't an issue we've dealt with (we read the manual), but it plagues millions of mongo databases that are just open to the world.
The query language sucks. Queries are JSON documents. Here's two queries, one for Mongo and one for MySQL.
[code]
db.items.find(
{
"$or":[
"Bot": 15,
"$and": [
{"User": 17},
{"$or": [
{"Price": {"$ne": 15}},
{"TimeUpdated": {"$gt": 1600}
]}
]
]
}
[/code]
[code]
SELECT * FROM `items` WHERE `bot` = 15 OR (`user` = 17 AND (`price` != 15 OR `time_updated` > 1600))
[/code]
Which is easier to parse? It's kind of a nonsense query -- I don't know if you'd ever get useful data with that sort of query -- but it demonstrates the point.
Oh, and if you want to do meaningful aggregation on a collection, you sometimes might want to use the mapReduce functionality. What's mapReduce? Why, it's a way to map data in a collection together (grouping it), and then [i]use a javascript function to reduce each array of mapped data into one document per-group[/i].
Seriously. You need to write a JavaScript function. To do some pretty basic queries. There are very few applications where that javascript function is easy to maintain.
All in all, MongoDB is useful for [i]some[/i] use-cases. In some areas, it can be more useful than MySQL. For instance, you can just chuck the JSON response from an external API into a document, and then run (reasonably) fast queries on it. No need to worry about schema or transforming the data. We, however, fit none of the use-cases for MongoDB.
Some of why MongoDB was a mistake is because we're using it incorrectly. Mostly, however, it's because it's just terrible for most of the things we want to do.
[editline]11th June 2017[/editline]
Oh, and MongoDB has no transactions.
The database engineer i worked on a project with, just kinda disregarded it as 'appealing because of simplicity'. I don't know how biased that opinion is, but it does seem like if you really know what you're doing you can mold PostgreSQL to fit about any usecase.
[QUOTE=Cold;52341180]The database engineer i worked on a project just kinda disregarded it as 'appealing because of simplicity'. I don't know how biased that opinion is, but it does seem like if you really know what you're doing you can mold PostgreSQL to fit about any usecase.[/QUOTE]
Yeah, I know Postgres has some really nice JSON functionality, which can fit a lot of use-cases that people would use Mongo for.
MongoDB may seem simpler on the surface, but once you start using it it's convoluted and really annoying to use. You pay a price, and you don't get much simplicity for it.
[editline]11th June 2017[/editline]
The one thing I will say is that it has really nice replica / sharding support built-in, but even then, every shard has to be a replica set, and there are config servers that also need to be a replica set for sharding. It's weird and convoluted. That seems to be a theme.
I've never liked working with relational databases that much, I'm using sqlite on my Android app and every time something SQL comes up I lose a lot of motivation so I have to push myself through it. The tables add up pretty quick, throw in one many to many relationship and you've already got three tables.
I feel like it can get out of control pretty easy if it's not just a basic use case.
[QUOTE=reevezy67;52341219]I've never liked working with relational databases that much, I'm using sqlite on my Android app and every time something SQL comes up I lose a lot of motivation so I have to push myself through it. The tables add up pretty quick, throw in one many to many relationship and you've already got three tables.
I feel like it can get out of control pretty easy if it's not just a basic use case.[/QUOTE]
I'm assuming you're writing queries manually. An ORM library is immensely helpful if you can sacrifice some query speed.
[QUOTE=reevezy67;52341219]I've never liked working with relational databases that much, I'm using sqlite on my Android app and every time something SQL comes up I lose a lot of motivation so I have to push myself through it. The tables add up pretty quick, throw in one many to many relationship and you've already got three tables.
I feel like it can get out of control pretty easy if it's not just a basic use case.[/QUOTE]
Maybe you're using the wrong tool for the job, then?
It seems you're just trying to store data generated by that single app instance, locally on the device. I don't know if SQL is the best tool for that.
Storing -- and retrieving -- data generated by many users, that can be very inter-[i]related[/i] is a very good use case for [i]relational[/i] databases.
[QUOTE=geel9;52341006]Its NoSQL design interested us; not having to worry about a schema seemed very handy.[/QUOTE]
This is a trap that many people don't seem to realize when trying out MongoDB. You can fill the collections with garbage and you might not even notice anything is wrong until you actually need to work with the data you've accumulated.
A database I worked had a collection with a timestamp for each entry, except that in some cases the timestamp would be a float and sometimes it would be an ISO 8601 encoded string but as a "string" field type, making any kind of date range queries impossible without converting them to MongoDB's actual Date field type first. So, essentially the "worry-free" schema-less design meant I had to write a manual migration script to fix the fields and fix any insert/upserts in different parts of the application to use the correct field type.
[QUOTE=Matoking;52341285]This is a trap that many people don't seem to realize when trying out MongoDB. You can fill the collections with garbage and you might not even notice anything is wrong until you actually need to work with the data you've accumulated.
A database I worked had a collection with a timestamp for each entry, except that in some cases the timestamp would be a float and sometimes it would be an ISO 8601 encoded string but as a "string" field type, making any kind of date range queries impossible without converting them to MongoDB's actual Date field type first. So, essentially the "worry-free" schema-less design meant I had to write a manual migration script to fix the fields and fix any insert/upserts in different parts of the application to use the correct field type.[/QUOTE]
Absolutely. Along with that, most applications using MongoDB [i]probably have a schema[/i]. It's just not codified into the database.
So what ends up happening is you're checking if a field(s) even exists [i]every time you fetch a document.[/i] You're putting MORE WORK into your code because only half of your stack is aware of your schema.
[QUOTE=geel9;52341273]Maybe you're using the wrong tool for the job, then?
It seems you're just trying to store data generated by that single app instance, locally on the device. I don't know if SQL is the best tool for that.
Storing -- and retrieving -- data generated by many users, that can be very inter-[i]related[/i] is a very good use case for [i]relational[/i] databases.[/QUOTE]
Yeah, I thought about serializing in to a file to avoid the hassle, but it would have a lot of duplicated data, I want it to handle thousands of entries, might need to bulk import at some point, and might have a ton of r/w operations.
I'll put aside my dislike for it for the efficiency/reduced size it gives.
[editline]12th June 2017[/editline]
Also, really long term, if I decide to add cloud storage or something it'll be a copy paste job to get it working.
[QUOTE=geel9;52341292]Absolutely. Along with that, most applications using MongoDB [i]probably have a schema[/i]. It's just not codified into the database.
So what ends up happening is you're checking if a field(s) even exists [i]every time you fetch a document.[/i] You're putting MORE WORK into your code because only half of your stack is aware of your schema.[/QUOTE]
^this is exactly what happens at work atm for me... decisions which frameworks and databases are going to be used are made like this:
"i used it once at university"
"i heard about ..."
"lets try this i found it on google"
"... is also using this"
my inner developer is like: seriously ?
Got myself Raspberry Pi Zero W, the end goal is to remote control my radiator so that I can turn it on before I get back to my apartment. I just finished putting it all together, just need to finish the software now.
[t]https://images.discordapp.net/attachments/257291143102988289/323586977603518474/20170612_000344.jpg?width=508&height=678[/t]
[url]https://my.mixtape.moe/qqlqgk.mp4[/url]
[QUOTE=geel9;52341006]Are you referring to data loss or the website code breaking?[/QUOTE]
Yes, or yes, or the script/migration that you're using breaks, and you have to debug it. Please don't say that "it won't break," because things break, and having a backup plan is always a good idea. Lord knows how much money I've lost not doing that.
[editline]11th June 2017[/editline]
[QUOTE=geel9;52341292]Absolutely. Along with that, most applications using MongoDB [i]probably have a schema[/i]. It's just not codified into the database.
So what ends up happening is you're checking if a field(s) even exists [i]every time you fetch a document.[/i] You're putting MORE WORK into your code because only half of your stack is aware of your schema.[/QUOTE]
Also yeah, most codebases have [i]some[/i] sort of schema, whether or not the developer expects it. It's a free optimization, so I'm not sure why people like no-schema no-SQL databases.
[QUOTE=Ac!dL3ak;52343292]Yes, or yes, or the script/migration that you're using breaks, and you have to debug it. Please don't say that "it won't break," because things break, and having a backup plan is always a good idea. Lord knows how much money I've lost not doing that.
[editline]11th June 2017[/editline]
Also yeah, most codebases have [i]some[/i] sort of schema, whether or not the developer expects it. It's a free optimization, so I'm not sure why people like no-schema no-SQL databases.[/QUOTE]
We have auto-verification scripts that ensure every piece of data was migrated over. If it goes wrong, we can just re-initiate the migration; as I said, it doesn't take long at all.
We also have a full database backup collected every 12 hours, along with a backup of the "vital" collections (users/orders/items/etc.; anything important really) taken every 10 minutes. Both are uploaded to AWS Glacier, and yes, we've tested them.
[QUOTE=Ac!dL3ak;52343292]Also yeah, most codebases have [i]some[/i] sort of schema, whether or not the developer expects it. It's a free optimization, so I'm not sure why people like no-schema no-SQL databases.[/QUOTE]
Schemaless is useful when you have very large datasets and you cannot afford the downtime to run migrations upfront. It does mean you have to support each and every version of your schema forever, but... when you're Twitter, you don't really have a choice.
And that's the context NoSQL databases in general should be viewed in. Useful when you're at least as big as Twitter, or a Silicon Valley startup whose business plan dictates that if you're not at least as big as Twitter one year after launch (or get bought out by someone who is), you might as well declare bankruptcy. Under such extreme constraints, you [I]have [/I]to trade in ACID and normalisation and your programmers' sanity for performance in general and horizontal scalability.
[QUOTE=geel9;52343360]We have auto-verification scripts that ensure every piece of data was migrated over. If it goes wrong, we can just re-initiate the migration; as I said, it doesn't take long at all.
We also have a full database backup collected every 12 hours, along with a backup of the "vital" collections (users/orders/items/etc.; anything important really) taken every 10 minutes. Both are uploaded to AWS Glacier, and yes, we've tested them.[/QUOTE]
Ok, but what happens when you have to pause the script for an indefinite amount of time? Are you going to leave the site down, or are you going to continue at a later point?
[editline]11th June 2017[/editline]
[QUOTE=DrTaxi;52343922]Schemaless is useful when you have very large datasets and you cannot afford the downtime to run migrations upfront. It does mean you have to support each and every version of your schema forever, but... when you're Twitter, you don't really have a choice.
And that's the context NoSQL databases in general should be viewed in. Useful when you're at least as big as Twitter, or a Silicon Valley startup whose business plan dictates that if you're not at least as big as Twitter one year after launch (or get bought out by someone who is), you might as well declare bankruptcy. Under such extreme constraints, you [I]have [/I]to trade in ACID and normalisation and your programmers' sanity for performance in general and horizontal scalability.[/QUOTE]
Interesting... I work at a place that's almost required to use a schemaless datastore, and so I know of a decent number of reasons to use schemaless, but migrations never crossed my mind. That's helpful, thanks!
[editline]11th June 2017[/editline]
Although I wonder how something like CockroachDB fits into that.
[QUOTE=Ac!dL3ak;52344571]Ok, but what happens when you have to pause the script for an indefinite amount of time? Are you going to leave the site down, or are you going to continue at a later point?
[/QUOTE]
...what?
The data still exists in MongoDB. It's not like performing the migration deletes the data. If the migration script doesn't work, we just won't activate the branch.
Derailing interesting database discussion (interesting primarily because its all so new to me) with my awful graphics experiments... buuuut, I got LOD stuff mostly working!
[t]http://i.imgur.com/J84vyBb.png[/t]
There are still some gaps when it comes to inter-LOD stitching, but the intra-LOD stitching works great and tiles at the same detail level stitch well. This is despite the obviously artifacting/repeating noise I'm using right now, too:
[t]http://i.imgur.com/2dcdvKl.png[/t]
I'm honestly super pleased with this and myself! I was beginning to feel like giving up and that this was outside of my scope, but I tried to remain calm, brewed a cup of tea, got out my engineering paper and began drawing out the grids and what I thought the various problems were. Spending more time checking the math - and fixing an assload of silly mistakes (e.g using size_t when I needed a signed int) - seems to have been the trick. There's still loads upon loads to do, but I feel that most of the hard work is behind me. I need to thread the generation somehow though, as it does freeze/lock-up the rendering thread right now (unsurprisingly). That and I need to improve how it looks, so debugging it is less of an eyesore
Soon, though, planets! I got cubemap-to-sphere mapping of mesh and noise data working last time I tried planet rendering, but could never get LOD stuff figured out and that was what stopped me last time. This time, I believe I can pull it off.
[editline]edited[/editline]
I expressed the sentiment on my mug many many times, admittedly :v:
[t]http://i.imgur.com/m4NaRsA.jpg?1[/t]
[QUOTE=geel9;52344628]...what?
The data still exists in MongoDB. It's not like performing the migration deletes the data. If the migration script doesn't work, we just won't activate the branch.[/QUOTE]
I'm talking about the end result for the users - I understand that the data is in the database. Let's assume that you set a 2 hour window for the migration. Let's assume something breaks, as things are wont to do. It will now take 3 hours for the migration. Do you attempt to fix the issue, keeping the site down? Or pause the migration, bringing the site back up? What if it takes 5 hours? 8 hours?
Migrations are super difficult, and things that you didn't know could go wrong will likely go wrong.
[QUOTE=Ac!dL3ak;52345021]I'm talking about the end result for the users - I understand that the data is in the database. Let's assume that you set a 2 hour window for the migration. Let's assume something breaks, as things are wont to do. It will now take 3 hours for the migration. Do you attempt to fix the issue, keeping the site down? Or pause the migration, bringing the site back up? What if it takes 5 hours? 8 hours?
Migrations are super difficult, and things that you didn't know could go wrong will likely go wrong.[/QUOTE]
In such an event we'd likely keep it down until the issue were fixed. Even 24 hours of downtime isn't a huge deal. It'd suck, but we're not going to have millions of angry customers here.
[QUOTE=geel9;52341006]-snip-
So, there are a few main issues with MongoDB. The first is kind of our fault -- we've been using it like a relational database. We have collections that have fields that point to IDs in other collections. MongoDB isn't really supposed to be relational -- that's the whole point. However, the alternative is to store data in multiple places .. (snip) .. Now whenever you edit a post, you have to edit it in multiple places. And if you edit a user's data -- like their name, name color, avatar, etc. -- you have to edit [i]that[/i] in multiple places. This seems to be the preferred MongoDB setup, and we don't like that, so we don't do it.
Also, MongoDB has no joins. Well, okay, it has joins. They're just one of the largest pains to use, and they're not incredibly powerful. In order to join two collections together, you need to use the aggregation pipeline (more on that later). This is a bitch, and AFAIK you can't do multiple joins in the same query (I could be wrong, though.)
-snip-
[editline]11th June 2017[/editline]
Oh, and MongoDB has no transactions.[/QUOTE]
These above are hands down the biggest reasons people should avoid MongoDB. I can't imagine a scenario where you would want this over a relational database (other than very very small projects).
Is there ever a moderate sized project where your data has no relations whatsoever? PS: I'm not scolding you, I just thought I would let others know that I agree with your decision, nobody should use a NoSQL database.
MongoDB is clearly not amusing anyone here. Let the man migrate his database to MySQL in peace!
Where MongoDB could be used though? Logging? I remember it having high write speeds and you could aggregate logs I guess
[QUOTE=suXin;52346102]Where MongoDB could be used though? Logging? I remember it having high write speeds and you could aggregate logs I guess[/QUOTE]
AFAIK searching stuff in mongodb is super slow and there exists way better DBS for logging purposes.
I think it's suitable for prototyping since you can get started very quickly and the schemaless makes it easy to adapt
[editline]12th June 2017[/editline]
Well, seems mongodb is indeed used for logging stuff, NVM then.
Why use Mongo even for logging, though?
Unless I'm confusing something here, InfluxDB is way faster for metrics/logging.
[url]https://www.influxdata.com/influxdb-is-27x-faster-vs-mongodb-for-time-series-workloads/[/url]
[b]EDIT:[/b]
We use it at work for a server status/metrics dashboard with Grafana
What are you working on? v67 - March 2017
