“(Question on Predictive Modeling” (not spoken into microphone)
Simon Burke, Xilinx: The first part of what you do is have a predictive model of what’s normal – what you should expect to see for all your design blocks. Once you get partway through the project and you get some confidence in that model, then you can start looking for places where your design team is starting to depart from the prediction.
Now it could be that they’re right and the predictions wrong or vice-versa, you don’t know. But the important point is you’re looking for those departures so you can go debug and figure them out.
Once you figure out that the team is departing and it’s a real departure, then you can start looking at what changed. Was it more complexity? Is it harder to do? Is there something fundamental about the design that is different? It lets you start tuning the prediction model to accommodate that change and, more importantly, what you really want to do when you see those departures is try to figure out how to apply more resources to that block to get it back on schedule.
To the first order: Why are they late? It doesn’t really matter. The fact that it is late, is the important point. And what you typically do, what we have started doing, is to cannibalize resources from projects that are on schedule, or blocks that are on schedule, to apply to the ones that are behind – so that you all finish at the same time.
It might seem counter-intuitive, but you know you’re already in a situation where you’re behind schedule. You want to minimize the impact of that. You use it in two ways. One is predictive in terms of what’s normal, and then you start looking for aberrations from normal, and figure out what to do about them.
I have a question on the scaling factors you used. Is there a way to learn those scaling factors? For example, based on design size, complexity of constraints, vis-à-vis technology, and things like how do you get these 1.2x versus 1.4x versus these numbers?
Simon Burke, Xilinx: What we’ve experienced is that we can put our designs into certain buckets. There are mixed signal designs, there are P&R designs, and there are more full custom designs. The process scaling factors for those tend to be somewhat different, but very consistent within that sub-type. So figuring out what you have in your chip and how to bucket them is the first step.
The complexity scaling is really, that every design can scale differently for that. What you’re looking for are pre-triggers or pre-predictors. If you’re worried about layout resources, you look at the schematic or RTL for the same design to figure out how much it changed from one project to the next.
You use those early estimates for what the complexity scaling would be, and then you refine as you go forward.
So yes, you start off with an estimate which is why you don’t want to predict tape out in the beginning of the project. It’s a nonsense number. What you want to do is, as you go through the project, you’ll see that the variation of those predictions start to settle down. If you do them every week, they’ll be swinging all over the place and then kind of settle down to a common point.
That’s when you kind of know you have the buckets right and you’ve got the predictions right.
And once that variation starts coming down, and your error bar starts to close down, then you can start using it as a predictive model, and start looking for departures from the predictive model and what that means for tape out. It’s work – you don’t just push a button and it happens.
Audience Member: How much data do you need to do what you talked about? And are the models flexible or do you have to tune it after you know you go through a couple of cycles or something to adjust those models for new predictability or new things you’re doing?
Simon Burke, Xilinx: We track every single file check out and single check in, by whom, when and every single license check out and license check in so it’s a fairly significant amount of data. We do maybe 5000 check-ins a week. And each of those check-ins can be either one cell or 100s of cells. So the volume of data is quite significant from that perspective. In terms of tuning and iterating, what we did was:
- Went through one project where we just kind of collected and didn’t even try to predict, because it was pointless.
- Once you have that side data, the second project is where you start predicting. But you keep it private so you can correlate whether your methodology works or not and you tune it. The second project tells you that you can predict the right answer based on what actually happens.
- Your third project is where it actually starts to be useful, and you can use that data going forward.
So yes there is a learning tuning cycle. The key for us was we wanted to be able to intercept the raw data to present the data with a programmable language that we could tune and adjust and with a vision system – that’s Python. We can collect all the raw data, collapse it, create metrics, then scale and weigh them however we want to come up with that final answer.
Audience Member: How long did it take you to finally get to your final finely tuning, tuned model?
Simon Burke, Xilinx: The first project was at 28 nm and we collected data on that, pretty much. The second set of projects were in the 20 nanometer node, and also we taped out multiple projects.
- The first one our prediction was within probably plus/minus two to three weeks.
- In terms of tape out and the man-hours resource was has always been within about that 2 percent window of tracking.
- For the third project, our internal team stopped tracking man-hours because the answer we got from our system was on the money. So they just stopped doing checking because it was a lot of effort, and no value added.
- The tape out prediction closed down to about a two week window so plus/minus one week.
Audience member: Have you applied this to other than looking at tape out time to market? Have you applied it to bug tracking, issue tracking, verification, any other part?
Simon Burke, Xilinx: Our initial focus was really two important metrics:
- When are we taping out, can we have a more predictive model of that instead of reactive?
We obviously hire a lot of contractors, in additional to full-time, and we pay them. That has a big budget impact. So we want to know how many we’re hiring, for how long and where and who uses them. So those are the initial two metrics we were interested in getting very good accurate predictions for. We will deploy this into other areas of our design process.
We started this project off to try to predict tape out and where resource was going from a project level. We have to do that at a design level so that the project roll-up kind of makes sense. What we found was a lot of our design teams want to know how much resource they’re using, whether they’re on track, and where things are going. So we were getting a lot of requests from them:
“I don’t care about the rest of the chip just tell me about my stuff”.
And we went down the path doing a presentation layout. Silicon companies tend not to do those things well. It’s not something we’re either good at or have a lot of interest in to be honest. So you end up giving them the raw data and spreadsheets and they figure it out.
So we did find these design teams were very interested in being proactive about their own resources and their own tracking and looking for aberrations early for the same reasons we were doing this.
The sooner they know about problems the sooner they can delete it. But they want to be able to do specific drill downs: they just want to know about this IP or this set of IPs or this variant because they own the subset of data not the complete set. The charts I presented allow you do that kind of drill down. I really care about the project levels and how it rolls up. But they want to know:
- Just for these two IP’s how do I compare my last project to this project?
- These two IP’s are supposed to be the same, how do they compare?
- I own this one, what is this one doing?
So [Envision] lets you do that kind of low level drill down into specific data types and specific IP’s. And then look at the subset data just for that stuff. A lot of design teams are using it from that perspective. They want to know their part of the world, and not the whole context.
It’s kind of interesting that what came out of the big picture actually turned out to be something for the design team from a very low level specific perspective. I skipped through them because of interest of time. (And the data is generic so when you read it, if you are Altera don’t try to assume what we’re doing it’s just made up data.)
This gives you a feel for the type of reports you can do very efficiently and quickly. And that’s really what’s important. A lot of them are customizable. Again you can tune the data via Python so your end user doesn’t even have to be a programmer. They just see the reports they want.